I think it's not as hard as all that.
Fer'instance, Bards can cast in Light armor. As can Warlocks. There's feats that mitigate it a pretty big chunk at a time, and a variant feature in Complete Mage that enables Light Armor casting for Fighter/Mages, and a half-score of PrC's that reduce the ASF%, too.
Part of it is flavor, like the d4 HD instead of the d6 HD. Part of it is achetype -- Wizards should have no need of mundane protections like armor and swords! Part of it is balance -- a low-AC, low-HD, low-BAB niche needs to be filled by something with amazing firepower.
So I'm pretty happy with it as it stands. Those who play against achetypes are given the tools to do so, but the classic image of the wizard "armed only with his wits and the power of the universe!" is kept. As it should be. If it was too easy to get around, we'd see a lot of armored wizards who think they can stand in the front lines because they have Monklike AC, when they're really just leaves in the wind.