To use a somewhat dramatic analogy: Folks were trying to build flying machines for centuries. That should have strongly suggested to the Wright Brothers that flying machines were a non-starter.
The mechanics I've been mentioning are all relatively new to RPGs. Which means that people are thinking up new stuff all the time! Why stop trying when there's always new things to try?
And none of the "new mechanics" have actually made it mainstream. Further, as seen up-thread, they are often disliked.
I also submit that no, folks were not trying to find "narrative options" for non-spellcasters for D&D for decades. Folks have been doing many things, but looking at it as "narrative options" is a pretty new concept. They have, instead, been trying to find ways for fighters to dominate combat more, and have skills outside combat, but those are only a couple of things one can do - folks may have been trying to improve the fighters, but they have had some metaphorical blinders on. As time goes on, and more games have more novel approaches, our palate of potential solutions to problems grows.
You are welcome to try. I can't imagine it being anything but a waste of time, but hey, your time. On the other hand, focussing on giving Fighters narrative options (no success so far) blinds you to the possibility of explicitly not giving them narrative options, which has been historically decently successful.
Because, as we all know, in 1e and 2e magic users were never forces in combat! Fireballs and lightning bolts are sacred cows in the game because they are pretty, but they never had any real weight in a fight! I'm sorry, that was my sarcastic voice, and I should not have used it. But the point remains - your memory of pre-3e gaming does not match mine. Wizards in earlier editions had low hit points, but were often crucial to winning fights.
Kinda/sorta/not really. On one hand, you have 1/2e Clerics. A powerful class (not a heal bot! the lack of heals at spell levels 2 and 3, and the lack of Cure Wounds scaling saw to that), with a decent defense, but no offense to speak of. Then you have Wizards, with "amazing" offensive spells... that could be matched by a Fighter in a round or two. And the Wizard got actually just-about DnD Next numbers of spell slots. Wizards weren't helpless offensively (although Fireball and Lightning Bolt were both... difficult.. to deploy if played straight)... but were completely helpless defensively, thanks to not actually having any HP. A 1e/2e party that tried to engage in combat without a Fighter (or Fighter subclass) was bonkers. What Wizards *did* have was enough of an offensive punch to contribute on final boss fights.
If that's the way you have it, then players of casters are generally bored in combat, and players of non-casters are generally bored out of combat. Being bored stinks. Writing off a player for large sections of the game is probably not a desirable design goal.
Actually, it is a good design goal (as long as the write-off isn't total). What we got in 3e was lots of combat, which resulted in Casters taking lots of combat spells, which meant adventure designers upped the fight difficulties, which resulted in casters doing full burns, which resulted in the complete sidelining of Fighters, and the 5MWD. And boring modules, because the Casters could contribute (heck, dominate) combat, while the Fighters couldn't contribute there.
By having some characters be not-good at out-of-combat stuff and some characters not-good at combat, you (a) force module design to be more interesting (you'd see a lot less set-piece battles, and more plain "how do you get past this") and (b) make your whole game-balance problem much, much easier. It also forces players to play outside of their character's specialty, forcing creativity.