I agree that Gandalf is not distinguished solely, or even primarily, by his spell-casting. He is distinguished by his knowledge, and his ability (in virtue of that knowledge, plus his personality) to lead.
As I posted upthread, it is possible to have a FRPG in which a wizard PC has that sort of capability, and hence gets to feel "wizardly" in play without casting a spell every moment. But it requires mechanical devices that D&D has, on the whole, tended to eschew.
Speaking of literature, it's difficult to emulate popular characters in a D&D game, and I don't mean mechanically. Gandalf was a plot device, more or less, the DMPC who did very little on screen and basically kept the party focused on the goal of destroying the Ring. He provided knowledge that the party could not reasonably have, he provided "DM fiat" when it was necessary and was generally more of a story telling device than an actual character.
The problem with story telling devices is that we've all played with people who've run DMPCs, and we all know they come in 3 varities:
The "normal joe" DMPC who just plays with the party.
The "Gandalf" who does very little, but serves to guide the party through the game.
The Dictator who steals the show and gets all the benefit of being the DM, with none of the drawbacks of being a character.
These aren't things you represent with mechanics. They're things you represent with role-play.
Which is often one of my gripes about people comparing this game or that game or any game to some quasi-related literature character and wondering why we can't have characters or classes that "look more like the source material". Because the "source material" is stories! They're not games. They're not defined by their rules and their mechanics. They're used to teach a lesson. That has been the basic purpose of stories for ages. Don't be the Boromir, be the Aragorn, deny power, stay loyal to your friends, know when someone in a position of power is wrong. Be the Frodo, stay strong in the face of danger, be willing to do what it takes, but don't forget your friends in the process. Don't be the Saruman, be humble, know when someone is using you, don't genetically engineer the ubermench.
D&D games don't do that. They don't teach lessons or shape minds, they are games. Hence, the characters in them don't look like the literature because they're two totally different things, it's like asking why a rock can't be like the ocean.
Anyway, at the OP: I think magic proliferates in D&D because people get obsessed with locking non-magical classes down with real-world rules, while the same people turn a blind eye to the reality-bending of magic. So when you ask a player: Do you want to be burdened by rules, or do you want to bend the rules over your knee and show it who's boss, what do you think most people are going to say? "I want my verisimilitude-obsessed DM to question my every move while Joe over there gets to
literally fly circles around me and drop magic hand-grenades on the enemy!"? baugh! It's no wonder magic proliferates when they can typically do everything a non-magic user can, and also wish themselves up a ham sandwich.