The fundamental philosophy of the game is really the issue here. Characters who cannot use magic are more or less bound to what we consider skilled people able to do in real life. At best, a Fighter or Rogue is a Hollywood action hero like one of the "Johns" (Wick, McClane, Matrix, Rambo), but it's a rare sight that a player doesn't want to try a stunt like "I'll slap the horse in the stables and make it kick my enemy" that doesn't immediately translate to "ok, take an action and make me a DC 15 Handle Animal check".
Magical spells, however, by their nature, can bend or break the laws of physics, like hackers in the Matrix. You can fly, phase, teleport, polymorph, shoot lighting bolts, create an army of clones, build a fortress out of rubble- the sky is the limit, because this is what fantasy gamers feel magic is and should be.
D&D attempts to close this gap by making magical spells limited in use, to justify their power. And if all spells did was buff, debuff, damage, or heal, that'd be fine. Fireball for 8d6 to fill most of a room? Sure, but you can only do it twice per day, and the damage might fail to kill a CR 1 Bugbear.
But how do you compare skill use or fighting ability to, say, "the power to turn a ruined old keep into a veritable fortress consisting of magical traps and magically locked doors" (Guards and Wards). You can't. While that power might be only useful once in a blue moon, the same way building a wall to surround a village with Fabricate is, when that incident occurs, you can exert this great narrative power in a short time, where your non-magical allies could only accomplish the same feat with hours of grueling work and many skill checks.
Because there really isn't any way to balance these abilities, as they are simply apples and oranges, orthogonal to one another, and the spellcaster can produce something from their bag of tricks without warning in the right moment, it's no wonder that the Wizard, who has the most deep and varied spell list, has been seen as a proud nail.
Your carefully built BBEG encounter? Trivialized due to a localized ice storm or exiling a demon to the X-Zone for a minute. Your fun detective story? Foiled by mind reading or divination. Even the threat of an army of zombies assaulting a village can be undone once you transmute solid earth into a quagmire of mud!
Magic plays by different rules, and that's exactly as intended. Look at 4e, when Wizards were probably the most balanced they have ever been. People hated it, because the Wizard "didn't feel magical anymore".
Players (and even DM's) exist who want the Wizard (and his spellcasting brethren) to have these kinds of powers. And many of these same people would tell you it's weird and unrealistic for a warrior to be able to leap 30' into the air to RKO a dragon out of nowhere!
So that's the problem with the Wizard; the expectations of the players, who want the class to be able to produce magical effects, large and small, both at will and in limited amounts, and solve all problems, from combat to setting up a campsite, with magic, while at the same time, wanting non-magical classes to be unable to do likewise.
