Once you've done 5 instances, you cannot enter anymore on the same account until a certain timer passes. I believe this to be implemented to avoid mass powerleveling or item/gold farming, however I think it's nonsensical to keep the same limit on raid groups, so I suggest removing this limit for 10-25 man raids.
What's more, I'd suggest considering to remove it for heroic dungeons too, as they cannot be entered multiple times.
P.S. If there's another reason for the limitation, would anyone from the staff answer why is it so?
It exists here because Blizzard made it that way. It could be to prevent people from farming something they aren't supposed to. Like mining inside instance or killing specific mobs and resetting it. Outside instances you would compete with other players and wait for respawn times, which would prevent abusing. When you farm something in an instance there is no competition and you can respawn everything yourself by resetting the instance.
It's 5 instances per hour. It's not that easy to hit that if you are doing dungeons properly. Except if you farm fos or pos and enter next instance after the dungeon just to get out.