Having actually played a brawling fighter that encountered a swarm (the Mul in Dark Sun Encounters), I feel that this is all a storm in a teacup. Sure, the Mul could have grabbed the damn swarm, but it had a damaging aura, which made that a pretty stupid idea. It doesn't make sense that a humanoid could grab a swarm of rats or bugs. It also doesn't make any tactical sense for the humanoid to do that, in many cases, so it balances out.
I mean, if you want to argue about common sense, it's possible for a small-sized creature (say, a halfling fighter) to grab a gargantuan creature and move it around. You could argue, "Well, it's all about which part you grab!", but even so, it makes little sense.