As far as I'm concerned, this should be an option, not a restriction.
This was one thing that killed me about 3.5, they gave the bard spellcasting in light armor, and then took MAGE ARMOR off their list!!
If a mage wants to use spells and magical energy to have armor, great!! It protects from ghosts as well. If they want to be wrapped in iron and suffer the armor check penalties and speed reduction, sure go for it.
I don't know why it needs to be one way or the other. Allow for armor, allow for magical means of protection, and let the player decide how his character will protect himself.