I agree with the fundamental point that numbers count for a lot when it comes to grappling. After all, a college heavyweight wrestler can trivially beat a high-school middleweight wrestler, who can himself trivially take down and pin an ordinary guy, but a half-dozen more-or-less ordinary guys can still dog-pile the heavyweight with some chance at success.But that brings me to my caveat. I guess four ordinary guys would be no chance of beating Brock Lesnar unless they were willing; they'd need sufficiently high levels of aggression or self-respect to hang in there if they took a big punch or kick, where most people would give up rather than stick to the game plan.
I don't think they'd dare, as you point out, and someone who's 265 lbs of muscle and very, very skilled at grappling isn't just a big, tough guy; he's strong enough and skilled enough to throw people around casually -- and to hurt them with his defense, even before he gets around to launching any legitimate offense.
How many 100-lb kids would it take to drag you to the ground -- if you were willing to hurt them?
Napoleon famously remarked that in combat morale was three times as important as physical factors -- what we'd call AC, hp, to-hit, damage, etc. -- but bravery and cowardice don't play into D&D combat very much.Adventurers can be as brave as they want.
Behaving like game pieces, or PCs, is exactly what real conscripts don't do, and I'd love to see the game help with that, instead of relying on an exceptional DM.