The only quests for which I refuse invites are the ones where you need to gather X things and they are not dropping for everyone. Which is pretty much less than 5% of all quests in the game.
Otherwise I agree with OP, there's litteraly no point in refusing the invites. Yeah, you get less xp per mobs, but quest xp is what makes you level anyway, and you go through the quests way faster with someone else just when considering that you kill mobs faster, and then there's the fact that they won't "steal" mobs from you.
No matter if you are leveling, or farming gold at 80, there's no point to not group when doing a "kill X things" quest.
However I rarely see people refusing invites for those quests, most have the same reasoning.