Because people are divided internally between emotion, instinct, and logic. According to most psychology studies most people are higher on the emotional scale than the logic scale. Instinct is harder to quantify but since a lot of it is programmed for cave man days and we've advanced way faster than we've evolved it causes reactions that don't always make sense in modern life. I don't remember the exact breakdown but I seem to recall there are 3 or 4 people who process emotionally vs every 1 who processes logically. I'd guess that most of those strategic // effective players who get so unhappy with what most people consider immaterial things are stuck on the logical side where they've been in the minority their whole lives and spend a lot of time trying to figure out why humans are so messed up.Agreed. And on top of that, TTRPGs are much less competitive than MMOs, which make concerns about efficacy and utility even less of a driver of people's choices.
The answer to the more detailed question "In 5e, are high-level spellcasters' contributions so useful that they trivialize a high-level noncaster's contributions" is pretty obviously a "Yes". I will definitely prefer the bladesinger who can send somone to Limbo with a touch and then attack with his ice golem simalcrum that's polymorphed into a dragon than the fighter who can attack with his hand crossbow 9 times.
Trying to determine if that truth stops people from playing noncasters at high levels doesn't make caster superiority less true. All the numbers show are that plenty of people are happy making nonstrategic choices, even at high levels, in the name of playing to a preferred concept. And given that it's impossible to "lose" a modern RPG, why wouldn't making the choice of concept over effectiveness be a common choice?
As one of the logical ones I have to point out it's not them that's broken it's us... We don't fit...