I always note that the chance of spell failure increases dramatically with every pound of iron, steel, or bronze the MU is carrying, due to the conductive resonance of the metal.
Which is why the magic users wear leather or dragonscale armor. Anytime they need a non-metal solution for something, they just ask the druid what he normally uses.
I quite like the option used in Skyrim, where spellcasters can wear whatever they like, but the best bonuses to spellcasting are the enchantments woven into robes.
What's the explanation for why you can't wear the robe under (or over) the armor? A video game is slot based. But in a system where you've already removed armor proficiency rules, what's to prevent it?