Absolutely not, assuming the nature of armor hasn't changed much. ASF penalizes characters more than necessarily for adopting an suboptimal character design.
Light armor is clearly not a big deal, since 1st level spells simulate it and bracers of armor +2-+3 are pretty cheap. If a character is proficient in light armor, they should be able to wear it.
Medium and heavy armors come with their own penalties: great weight, ACP, reduced movement. If you're slowing yourself down on the battlefield, you had better have the HP to survive getting hit in melee. You also need a high strength to wear it, but if a Wizard puts their second high score in Str then they're probably sacrificing Dex or Con.
Finally, the character that really need it--fighter/wizards--are taking enough of a power hit in their spells to balance the armor. The only thing the ASF does in this case is to force characters to spend big bucks on twilight mithril chain shirts instead of a normal breastplate. How does that help the game?
IMC, having proficiency in armor is enough to remove ASF. I don't have any armored wizards running around!