I always note that the chance of spell failure increases dramatically with every pound of iron, steel, or bronze the MU is carrying, due to the conductive resonance of the metal.
Which is why the magic users wear leather or dragonscale armor.

I quite like the option used in Skyrim, where spellcasters can wear whatever they like, but the best bonuses to spellcasting are the enchantments woven into robes.
What's the explanation for why you can't wear the robe under (or over) the armor? A video game is slot based. But in a system where you've already removed armor proficiency rules, what's to prevent it?