I don't like either ASF or a hard restriction, but I think armor should penalize magic unless you have some ability that makes it easier.
Like, you have disadvantage on your casting roll (casting rolls should be a thing) if you're casting in armor, unless you have a class or a theme or...