I completely agree with this sentiment, and this is why wizards are so much more powerful than fighters at high levels. And it doesn't even have anything to do with combat necessarily, but everything to do with the fact that "magic is needed for everything!!"
Creatures are invisible, need see invis. Need to teleport to go anywhere, divinations to know where to go. The fighter (and all martial classes really) have to spend the first good chunk on an adventure letting the wizard get his spells ready, let him do his divinations and figure out where to teleport and cast certain buffs, etc etc. The amount of the time the game focuses on spellcasters increases markedly at high levels....and no amount of raw damage by the fighter is going to change that.