D&D has always heavily abstracted stuff in order to be a playable game (to hit rolls, Armor Class, HPs, etc etc etc).
Often, that isn't a problem. In fact, the design is typically championed for what its trying to do. But it does sometimes arbitrarily become a problem, typically when people want HP to be meat or want D&D to be a granular process sim game (or at least parts of it to be).
The problem is when those two positions persist simultaneously (eg martial characters must be bound by earthly physics and spellcasters can just do whatever) and that persistence becomes (a) obviously incoherent and inconsistent and (b) (at least for me) a contributor to balance issues (and we circle right back to play functionality).
Sure, I agree with all of that.
I just don't feel like 4E was particularly controversial because it leaned in to that abstraction - it was controversial for more specific reasons, I think the biggest of which, is often little discussed, that being that it was 100% incompatible with a hugely popular edition people had invested in incredibly heavily, as had 3PPs. As such
@pemerton's reference to 4E was presumably about something else, or is invalid.
Put it another way - if 4E hadn't leaned into abstraction any more than, say, 5E or 2E (so much more than 3.XE, which was the least abstraction-oriented), but had been equally incompatible with 3.XE and 3PP products for it, and had equally had terrible marketing, and used jargon and approaches that put people in mind of MMORPGs (however accurately or inaccurately), would it still have been controversial?
I say absolutely it would. I say it would have had the same fate - or a worse one.
So is your answer to my question "just because"?
Maybe that is the answer, but in that case it doesn't seem a very good one.
What question?
My question is "verisimilitude to what?" You need to answer that I think. You clearly have something in mind. It is unclear what it is.
But before we start talking about "solutions" I'm still interested in interrogating the question. What is the principle of design that is at work in the current system?
Gamist balance is the principle I see, given the evolution of D&D from wargaming.
Also, at a certain point, you may need to allow that the main design principle is inertia, and that adding a new subsystem to "simulate" something in a highly abstracted game like D&D requires overcoming a fair amount of inertia, so needs some kind of further justification, like "this is necessary for balance" or "this will make the game more fun".
A lot of D&D's sacred cows are simply poorly-considered decisions made by inexperienced designers operating a near-vacuum design-idea-wise (rather than rich soup we have today). It may well be that this is simply how Gary saw Vancian magic working, in his mind, that it would go wrong if someone hit you, but that he didn't envision wizards just screwing it up otherwise.
The only edition which did have this was 3.XE, which simulated it if you wore certain armour (spell failure % or whatever it was called).