Yeah basically, the argument for verisimilitude is a bit flawed here, as player characters of any height can take on creatures that should outmass them to a ridiculous degree; a hill giant's club should send a human fighter in full plate flying through the air ala Skyrim, and their greatsword should bounce off a big ol' dragon's dense body structure like it was a Nerf weapon.
The 3.5 approach of downgrading offensive punch for small creatures by giving them greater accuracy and defense was perfectly acceptable to me (the bigger something is relative to you, the easier it is to hit, and the smaller profile you present in combat), but 5e doesn't like granular changes to begin with, let alone the effect on bounded accuracy if say, Halflings did d4 less damage but got d4 bonus to hit on every attack, which is why Enlarge/Reduce only alters damage.
And it's not like the racial abilities of Gnomes and Halflings are significantly better than anyone else, which is another way you could balance out the disadvantage of being Small, if there are no advantages allotted to it.
Anyone, large or small, being a viable adventurer and able to perform heroic deeds and fight deadly monsters is a large part of D&D's DNA, not being a simulated reality. So we have a lot of strange things we're intended not to look at too closely, like hit points, healing surges, the ability of dragons to fly or exhale lightning, and so on.
Some people argue that we shouldn't excuse one point of fantasy because other points of fantasy exist, but I think they're missing the larger picture- D&D is not grounded low fantasy. It's the kind of game that lets Conan defeat gorillas and lions with his bare hands or any random weapon he picks up, often with less armor than he'd like to have, and not only live to tell the tale at the next tavern he stops at, but win.
Now back to the OP, honestly, given the huge advantages flying foes have over most adventurers, them being susceptible to being knocked to the ground to "fight fair" doesn't strike me as a problem at all, far better than telling your big plate wearing He-Man that he better bust out a longbow or javelins and pray something gets in range of him (and that he can hit the broad side of a barn if his Dexterity is significantly lower than his Strength).
