So ok, let's say you're right, and the reason 5e is popular is because it is the way that it is. Maybe what we should be discussing then is why:
Let me start by saying that a bunch of what you are writing as your discussion questions are necessarily tilted to get the results that you want. But sure, I will try and in good faith provide you answers to your questions; note that these are my responses, and these are possible answers, but I do not claim that these are necessarily
the answers.
-More base classes would be bad for the game.
Because D&D is for the people. That means it needs to be easily accessible and grokable. As it stands, there are 12 (13 if you count the artificer) full classes in 5e, and innumberable subclasses.
That's already a lot in the
base game. Additional classes are always available via 3PP and homebrew. In addition, should the need arise, they can always make a specific new class for a setting/adventure/expansion.
But too much crunch kills editions. They want to keep the core simple.
-More love given to non-magical (sub) classes would be bad for the game.
I have no idea what "love," means. As it stands, the most popular two classes are the Fighter and Rogue - both of them non-magical. As should be obvious, additional "love" (whatever that means) would likely start to throw things out of balance.
-Making sure that players can't optimize themselves out of being able to participate meaningfully in a social encounter would be bad for the game (I know this is hotly contested, but it's my experience that if you want something from an NPC, there's really only three ways to succeed- skill check, magic, and convince the DM, and that last one, of course, carries the most weight).
As you note, this would be hotly contested. Why introduce something that is both hotly contested and also varies so much from table-to-table into the game? In other words, why be intentionally divisive in an area that very few people complain about?
-Allowing all classes to be able to perform incredible feats, not just magic users would be bad for the game.
This is just the same argument as the "love" argument, repackaged.
-Coming out and saying "yes, D&D worlds aren't real worlds, the laws of physics aren't our laws of physics, the humans are similar to but not exactly Earth humans, and by level X, it's assumed everyone can perform feats far beyond those of normal people even in their world" would be bad for the game (this one I know is even more fiercely contested, but most arguments against it come down to "that's not my preference", not whether or not it would be good for D&D).
This is just the same argument as the "love" argument, repackaged.
Without putting too fine a point on it, a lot of these points are basically, "Why not 5e, but 4e?" Sure. There are times I say, "Why not 5e, but 1e?" And other say, "Why not 5e, but where the heck are real psionics?"
There are small constituencies for various changes. But the point is- they are small constituencies. They are not looking backwards, but looking forward, and building upon a base that has largely developed during 5e.
Or, more simply- the primary market isn't you. And it isn't me. It's the people that joined up in the last ten years, and the people that will join in the next decade. It's okay that we aren't the primary market, because that means the game is thriving.