By the same token, if their hit points are that high it's unlikely the fireball's killing them outright either.
Question: does DoaM take into account the opponent's AC or other defenses? As in, if a 4e Fighter comes up against something it can't hit (or can hit, say, only on a 20), can that foe still slowly be worn down and killed by the Fighter constantly missing?
Well, yes; particularly if in your game a) fumbles are a thing, and-or b) a natural 1 always misses to no effect.
In 4e-5e where resource and hit-point attrition (sadly) isn't much of a thing as written, I'll concede that even if that 1-h.p. Orc is lucky enough to live long enough to get another swing in, and hits on that swing, it's not going to matter very much because those lost hit points are so easy to get back.
I don't, however, like or use that model. Recovery of hit points should either take time (by which I mean longer than a single night's rest) or be a drain on other resources (spells, usually).
As for casters, houserules are your friend. Make spells interruptable a la 1e such that it becomes essentially impossible to cast while in melee. Make spells take time within a round to cast, allowing greater opportunity for interruption. And most importantly, make them roll to aim their AoE's; once they've hit their allies by mistake a few times they'll be a lot less willing to fire at will.
And when the players of said casters complain, ignore them.
There's a fair bit to unpack here, so, I'll try a numbered list.
1. What if AC is Very High?
Well, this really isn't a problem is it? In 5e, with bounded accuracy, it's virtually impossible to have this occur. And, if it did occur, DoaM is not going to make any difference to the outcome - the PC's either run away, surrender or die. And, really, that's true in any edition. Even in 1e, it's nearly impossible to have enemies that you can only hit on a 20 after 1st level or so. And, anything that you actually did need a 20 to hit is going to obliterate you anyway, so, again, DoaM would make zero difference to the outcome. This is an outlier that really, I think, we can ignore.
2. Why not Change Casters?
Well, that's an option too. But, it is a MUCH more complicated one. And, the 1e rules for spell interruption, IMO, didn't work. Your MU should never be in melee anyway and, if he was, he was as good as dead. So, the MU constantly stood back twenty or thirty feet and lobbed daggers or the occasional spell or blew through charges on his hundred charge wand. The 1e rules just never made much of a difference.
And, your last point is telling: ignore the complainers. Well, why wouldn't we just ignore those complaining about DoaM? It's no different. And it's a LOT simpler to give a new ability to a character than force house rules onto a very large swath of the game. Your house rules would impact not only a single player, but, every caster player as well as the DM too. Now the DM has to futz about with aiming breath weapons and various other AOE effects that the DM's monsters can use. Which makes those monsters weaker - after all, you might miss part of the party with that breath weapon if you badly aim.
To me, it's far, far simply just to give fighters and fighter types a minor boost (and we are talking a REALLY minor boost - probably a couple of points of damage per combat) than add in very complex house rules.