I agree that the trade-off you describe is apt for survival gameplay. For "heroic" gameplay, where (I presume) players aren't particularly interested in keeping track of how much stuff they're carrying at all save in the most abstract sense, though, is that even a worthwhile trade-off to insist upon?
Apropos of annoying players, to my mind it's still better design if the game's rules aren't actively interfering with the kind of game players are interested in playing. That's not something that ought to have to be solved in session 0, except, of course, for the players to vote on or come to a consensus on what kind of game they want to play and then adding the appropriate variant rules.
This is, as far as I can see, plain and simply false insofar as you are asserting it to be some sort of "objective" property of RPG gameplay.
I am comfortable asserting that for the vast majority of players currently playing D&D, "How much gear can I carry right now?" is not an interesting question that they want to be made to answer through the game's mechanics. A more-or-less unlimited carrying capacity would, for those players, be genuinely better because they can focus their time and energy on those questions they do find interesting enough to answer through gameplay.
For those players for whom "How much gear can I carry right now?" is an interesting question, the way in which the game's ruleset and mechanics force them to answer the question also matters; some methods are going lead to more enjoyable gameplay by those players' lights than others. I am comfortable asserting that those methods are not going to correspond precisely to "realistic" encumbrance systems.
I'm also quite comfortable asserting that your preferences are very niche. Nothing wrong with them in and of themselves, but it would simply be a mistake for D&D to cater exclusively or even primarily to them.
Again, it seems to me that you are treating your own niche preferences as being "objective" qualities of RPG play - or, at the very least, you come across as not satisfactorily allowing for the fact that satisfying gameplay preferences you don't share is just as valid RPG gameplay as satisfying your own preferences.
I feel comfortable asserting that to the largest chunk of the D&D player base, their preference is either very casual "kick in the dungeon door and kill everything for its loot" gameplay, or character-driven gameplay with an overlay of strategic and/or tactical decision-making, and not for logistics-driven gameplay. "What dungeon are we knocking over in tonight's game session", "What goal is more important for me to pursue right now?", "What allies can I call upon in this situation, or what allies should I call upon?" or "Should I cast a spell right now and if so, what spell?" are interesting questions by such standards, while "Did I remember to bring a crowbar on this expedition or not" just isn't.
Spellcasting, then, is seemingly leading to more of the kinds of decisions players have to make that they find interesting and enjoyable to confront during gameplay. These decisions could relate to questions of resource management, but they don't have to.
Meanwhile, for players who do want to incorporate more mundane resource management and/or logistics-driven gameplay, it's not clear to me that a more "realistic" set of mechanics is going to do the trick. "How much stuff can I carry right now?" might or might not be an interesting question, but "How many pounds or fractions of a pound of stuff am I carrying that I have to spend precious table time computing?" is rather less likely to be.
Finally, bluntly put, challenge is, at best, only minimally "objectively" good for gameplay, insofar as gameplay that is not-at-all challenging, or that is consistently underwhelming as regards challenge, is likely to be boring. Beyond that, it is a useful element of gameplay when players find it interesting and enjoyable in and of itself. And, of course, even when players who are not Lanefan value challenge, it does not follow that a more "realistic" encumbrance mechanic is going to be an interesting and enjoyable way for them to experience challenge.