ASF
Most people agree that a wizard doesn't need armor. You get bracers when you can afford them (and no hurry thanks to Mage Armor). Even players of rogues, bards, and rangers like bracers because of the skill and movement advantages (druids, too, if the bracers are non-metal). In armor, swimming is difficult, it is loud, the max dex bonus limits the effect of cat's grace and the like, etc. Though it only matters in some games, wearing armor is often worse in town encounters and when you are dressed for bed.
Smart high-level characters rarely wear heavy armor (even fighters); full plate is still worn, but it is made of lighter materials (a.k.a. Dwarven Plate). High DEX characters almost always eventually opt for the elven chainmail or an equivalent at higher levels (light armor with only a +4 armor bonus, same as Mage Armor). Super-high DEX characters often have to do away with armor entirely to take advantage of their full DEX bonus. There are spells that improve your mobility in armor (such as fly), but any competent wizard can come up with improved mage armor as well. The difference either way is approximately one spell. The true test of whether armor is worn is in how your ability scores are arranged. Most characters with good DEX will have light or no armor.
As to flavor, I have always found that making the NPCs behave one way (establishing the flavor) works just fine. Let the PC wizards act how they want and the NPC wizards wake up in the morning, put on their robes, and cast mage armor (or an improved version--for those who don't know, the improved version formula is 3+level of spell armor bonus). Apprentices will not be tromping around their masters' houses in armor. Mid-level wizards will likely be found in town trying to make a living most of the time. Any wizard who has to don armor to do battle is low enough level that he/she is not a full-flavor wizard anyway. Throw on top of that the simple fact that STR is not as valuable to a wizard as other classes, and the armor will naturally be lighter (and if the armor is a chain shirt or the like, wearing it under robes will not detract from flavor, anyway).
In case my point is murky, since every other character may opt for light or no armor in order to advantage themselves, a wizard may end up doing the same. In fact it is likely that the highest level wizards (those on the order of Gandalf and Merlin) would not wear armor since they have sufficient armor bonus without it. Weaker or slower wizards, and Battle Mages, probably would. No big unbalancing going on here.
Nevertheless, I would recommend the armored casting feat for those who can't swallow the argument that armor is not unbalancing. Wasting a feat to cast spells while wearing armor should be pretty good incentive to keep wizards out of armor. Without the feat, a Concentration check is in order with the penalty equal to the skill check.
Though it doesn't "prove" anything, the wizard in my campaign chose not to take the armored casting feat. Incidentally, neither the wizard nor the ranger/rogue wear armor.
