the issue here is the same fundamental one as with martial vs magic problem solving: magic can do anything it likes in any way that it likes as we have no comparisons to hold it to in real life because it's magic whereas using weaponry is held to those boring old limitations of 'reality' and 'realism'
Don't forget how conveniently the "correct" way to portray magic often seems to make life easier for the caster!
5E was also designed (in theory anyway) to be simple. So you sacrfice a level os realism or simulation with that simplicity. Fortunately, house-rules and homebrews abound to add complexity if you want it.
It would be simple to ban arcane casting in armor rather than fiddling with proficiency. The line dictating what is acceptable crunch and what is not is pretty arbitrary. No reason why it always has to favor the magical classes.