How did you justify keeping tarts in a pouch and all the other silliness of material casting? How did you justify the rest of the magic system? Without knowing the framework you established in your game for the physics of magic, cannot say.RAW says that wizards can't cast spells in armour they aren't proficient in. I can't figure out how to justify this when it comes to spells that have only Verbal components. How do you explain that a wizard who is tied to a chair with his hands behind his back can still cast Misty Step, but relaxing in chainmail you aren't proficient in is too "distracting and physically hampering" to allow the same?
Anyone see any obvious pitfalls to allowing V-only spells while in nonproficient armour?
And where is "relaxing in chain mail" a thing? Are there lots of cases in your game where folks need to cast spells while relaxing in armor? In my games, folks tend to remove the heavy stuff when relaxing. Not so in yours? You guys got "full-plate party nights" goin' down?
Put another way, if our answers are "while relaxing, sitting around, not in combat it's ok, sure" is that an answer to your final question you are ok with - even tho your last question somehow, amazingly, inexplicably dropped the "while relaxing" part.
Or is what you are really asking about is casting while in these heavier non-proficient armors including combat but that was not as slanted-in-your-favor enough case? Too easy to see how the unfamiliar drags and pinches and such would make "distracted" seem reasonable?
Now, to address the question you asked instead of the question you staged... looking at it from game balance perspectives
Meta-magic subtle spell should then be raised to level instead of 1, at least. That removes the somantic components as well as the verbal and is not currently costed for it being a wide-spread AC boost.
The over-all limits for armour should be re-thought. If you allow spelling but still enforce the other penalties, you wind up make the actual fight-in-armor combat trained guys more impacted by non-proficient armor types than mages with much much less training. You might want to look at allowing simple weapons maybe to be used without penalty while in non-prof armor.
You might want to consider how wearing these armors in a combat situation might affect concentration checks. Remember a GM can require those round by round in circumstances that make it difficult. Would these qualify?
I am sure there are more potential issues, but these are the first thoughts I had.
The net result of the change will be swapping around a few niche edge cases and favoring different builds -especially if features "costs" are not adjusted to reflect the change.