My apologies.. I should clarify. It's not that I fail to see that such limiters can be useful.
It's that I fail to see why they are required, why they are assumed.
As to what fantasy means, by literal definition, it is the faculty or activity of imagining things, especially things that are impossible or improbable. It's damn near as "anything goes" as you can get. Grounding a setting with baseline physics in order to avoid adjudication of every single physical interaction characters have is a convenience, not a necessity of the genre. Allowing characters to occasionally break rules that are only really there for that convenience should not trigger the vigorous pearl clutching it seems to inspire. The truth is..there is no spoon.
As to the rules fictional humans need to abide by, there are many many examples of fictional humans who do not abide by the real world human baseline, with varying levels of explanation, from none to quite thorough biological discussion. D&D can easily operate at either end of that explanatory gradient, but honestly, it mostly functions better when you don't think about it that much.
Besides that, in D&D humans only represent one of many PC options. In many, maybe even most cases, one of a player's very very first choices is to depart from the human baseline. So we wind up with these really weird, kind of dissonant conclusions.
"So I'm a 400 year old dude who doesn't sleep, who maybe has a bloodline connected to the fey, but jumping over 30 feet horizontally is preposterous?" Uhhh..what.