I don't see it as "everything is the same, just colors changing." But magic and spell variety as a limiting factor in the mechanics of a sorcerer or wizard is a separate problem than the mechanical problems of the ranger. One can use existing spells to create new spells with slight adjustments or variations quite easily in order to create greater variety, the easiest of which is to change energy type. No, these are technically "cannon" or "RAW" spells, but it works, and can help a sorcerer or wizard achieve a particular theme while maintaining their mechanical benefits of their chosen class/archetype.
I just disagree that all elements should be equal, and that elemental substitution plays well. For me it undermines the idea of the different elements to begin with.
As for the monsters you miss from earlier editions, there's nothing saying you couldn't bring those back.
There isn't. I just used them as a hook for this topic.
I am not arguing that every class needs to be optimal in every situation. However, there is absolutely no major class ability that limits the use of an ability in the ways that favored enemy does. The only one that, as written, is more reliant on DM fiat is the Wild Sorcerers Tides of Chaos ability. Regardless of what you're fighting or what the situation, 90+% of all class abilities can be at least attempted. Only favored enemy limits the situations in which they can be used. Now mind you, this is not necessarily a bad thing. But it needs to be at best a minor ability that the ranger can do without, and should not be used as the focus for a capstone.
90% of all ranger class abilities can also be attempted most of the time. This is one of the problems discussing favored enemy. It becomes the "only" thing rangers can do, which is far from fair. If it should or should be the focus of the capstone, I don't know. I certainly find it inspiring to have an specialized enemy-hunter, who can, at the apex of their career, fight their favored enemy in a masterful way.
Finally, as to your point regarding how a player chooses their abilities or spells, it is perfectly acceptable to play a character that adapts to the encounters of the game. But what if you start the game at a a level higher than first and invest in abilities to fit your concept? What if you have a vision for how your character/hero develops their powers? What if their powers are not based on their adventuring experience, but rather hidden powers that are already defined but require certain stresses to unlock? There are many ways to play, and none of them are wrong. It is different if you have a fire sorcerer in mind that you want to play, but end up fighting a lot of monsters with fire resistance. There are abilities and feats that can allow you to continue to feel useful (Elemental Adept for starters, which can be used as a basis for another feat that allows one to damage creatures with an element despite normally being immune). These are easier matters to adjust, since a fire sorcerer can still cast spells against any creature or in any encounter so long as they have a spell slot. Additionally, as you mentioned, they can choose to use other spells if they know them. A ranger with favored enemy is in a different category. If this is supposed to be their defining ability around which a capstone is created, the ability must be useful even when not facing their favored enemy. Otherwise, theoretically they could go their entire adventuring career without ever fighting their favored enemy. Such an ability is completely reliant on the DM to play ball and throw the ranger so goblins or orcs or undead.
Of course, this is my perspective and how I view the design of 5e, and how I adapt the game to meet the needs of myself and my group.
I still just boils down to the so called "social contract" on the table. I have seen rangers in game in so many different editions, and most often than not they would get some sort of benefit against some specific group(s) of monsters. There was even a time when the player had no saying at all to what kinds of monsters their benefits would apply. And still, it played out well. I am used to sit on a table where everyone is on agreement to the general theme. It affects every character being built. It affects race and class availability, potential social challenges related to choosing one or other race or class, spell selection, combat styles for the warriors, skill selection as a whole, the favored enemy is just one feature in the list.
There are two key differences between the wizard and the ranger here.
The first is that the wizard is "on" by default and only turned "off" by these special encounters, whereas favored enemy is "off" by default and only turned "on" by special encounters.
The second is that the wizard is never really "off". He still has the opportunity to do interesting things with magic. The enemy may be immune to spells, but he can still haste the fighter and turn the rogue invisible, then control the battlefield with wall of stone. Whereas when favored enemy is "off", there's nothing interesting the ranger can do with it. Her numbers simply aren't as big. (By the same token, even when favored enemy is "on" it's relatively boring.) So for the wizard, golems are an intellectual challenge, whereas for a ranger, non-favored-enemies are just a bummer.
Well, yes, the wizard magic is "on" by default. And it is pretty much what a wizard gets by being a wizard in order to deal with their challenges. And still, their spell selection and strategies will vary and adapt according to the type of adventure or campaign they are into. And then the favored enemy is "off" by default, but in a campaign where players and the DM are on the same boat, it will be "on" more often than not, and even when it is not "on", it is not really the bulk of the ranger capabilities, not by a far margin.
Your second paragraph is really awkward for me. What if I wrote that the ranger still has the opportunity of doing interesting things with their several different class features, including spells, when their favored enemy is "off", but the wizard would have nothing interesting to do with their great repertoire of fire spells when facing fire-imune enemies, or of direct spells when facing golems? Then I could change the conclusion that for the ranger, non-favored enemies are just more challenging as they are not being able to use a one of their many class features, but they are far away from helpless as they have a huge amount of other features to use, while for that fire-enthusiast wizard, fire-imune enemies are a bummer.
I know my last paragraph is a real big stretch, and I am laughing as I write it, but wasn't yours also quite so?