This is, as far as I can see, plain and simply false insofar as you are asserting it to be some sort of "objective" property of RPG gameplay.
I am comfortable asserting that for the vast majority of players currently playing D&D, "How much gear can I carry right now?" is not an interesting question that they want to be made to answer through the game's mechanics. A more-or-less unlimited carrying capacity would, for those players, be genuinely better because they can focus their time and energy on those questions they do find interesting enough to answer through gameplay.
Chess would be genuinely better for many players if their pawns could all move like queens so those players could focus instead on winning strategies without having to worry about the limitations of their pawns; but they can't, and thus have to put up with the challenges of pawns being limited to what pawns can do.
I'm also quite comfortable asserting that your preferences are very niche. Nothing wrong with them in and of themselves, but it would simply be a mistake for D&D to cater exclusively or even primarily to them.
Needless to say, I disagree on both counts.
I feel comfortable asserting that to the largest chunk of the D&D player base, their preference is either very casual "kick in the dungeon door and kill everything for its loot" gameplay, or character-driven gameplay with an overlay of strategic and/or tactical decision-making, and not for logistics-driven gameplay. "What dungeon are we knocking over in tonight's game session", "What goal is more important for me to pursue right now?", "What allies can I call upon in this situation, or what allies should I call upon?" or "Should I cast a spell right now and if so, what spell?" are interesting questions by such standards,
Odd though it may seem, I agree with all of that except the bolded.
What's being wilfully ignored, however, is that
without in-game logistics none of these things can happen!
It's the same as hitting the road and focusing only on the interesting questions such as "which route do I take to get to town" or "where am I going on my road trip" while actively ignoring the boring-but-essential logistical question of whether the car has any gas in it.
while "Did I remember to bring a crowbar on this expedition or not" just isn't.
It might not be an interesting question but it's sometimes an essential one. See gas-in-car, just above.
Meanwhile, for players who do want to incorporate more mundane resource management and/or logistics-driven gameplay, it's not clear to me that a more "realistic" set of mechanics is going to do the trick. "How much stuff can I carry right now?" might or might not be an interesting question, but "How many pounds or fractions of a pound of stuff am I carrying that I have to spend precious table time computing?" is rather less likely to be.
I'll agree that tracking encumbrance is - ahem - cumbersome as it stands. The problem is that there really isn't anything else that's both simpler and equally (or more) realistic; and I don't like trading away realism for simplicity. I think the game long since did enough of that.
Finally, bluntly put, challenge is, at best, only minimally "objectively" good for gameplay, insofar as gameplay that is not-at-all challenging, or that is consistently underwhelming as regards challenge, is likely to be boring.
That last bit alone makes challenge far more than "minimally" good for play, as boring play very quickly leads to no play and part of the point of an RPG is (usually) to have people continue playing it.
Beyond that, it is a useful element of gameplay when players find it interesting and enjoyable in and of itself. And, of course, even when players who are not Lanefan value challenge, it does not follow that a more "realistic" encumbrance mechanic is going to be an interesting and enjoyable way for them to experience challenge.
Sometimes you just gotta take the bad with the good in order to get to a desirable end result. A hockey player, for example, might love playing hockey and yet detest having to tape his sticks or sharpen his skates before the game; but he's still gotta do those things.
That said, I really do think there's an encumbrance/carrying system out there (as yet uninvented) that is both simpler in practice yet realistic in fiction.