Actually, it's still fine. Schrödinger's Cat is only a mystery if you never look inside the box, but we look inside the box every time we determine the outcome of a roll.
Any time you draw attention to the nature of hit points, when you have a mechanic that depends on one definition over another, it opens the box. This applies to damage on a miss, rapid non-magical healing, lethal versus non-lethal damage, and a few other mechanics.
The nature of hit points is an ooooold argument. There's a sidebar in the 1st Edition DMG on the topic because it's an issue. They work best when they're the other hand of the magician, when the attention is on everything else but the hit points. Any time you start to pay attention the illusion breaks.
That's because hit points are a terrible system. The fatigue/ skill/ luck works some of the time but there's so many instances where it doesn't. But the alternative of static health that never increases, is equally problematic. Hit points are a system MacGyvered and held together with duct tape, gum, and faith. It works but it sure ain't pretty, and don't try and stretch the system in too many directions.
If the DoaM was not enough to knock the target out, it was fatigue.
There are a couple problems.
The first is damage types. For example, if you attack with a flametongue or frostbrand then you're still dealing energy damage. It's not generic fatigue damage but fire damage or cold damage or poison damage. Which matters a lot when fighting things with resistance or regeneration, especially the troll.
Damage on a miss dramatically changes a troll fight. Missing with a flaming weapon (even a torch or improvised weapon) is more effective than hitting with any other type of weapon.
The second problem is one of narration. A hit, if hp is energy, is a near miss that causes the target to tire themselves out deflecting or avoiding. A DoaM miss, if hp is energy, is a near miss that causes the target to tire themselves out deflecting or avoiding. They're the same thing in the world despite having opposite expected results. That's not
If the DoaM was enough to knock the target out, it was physical damage of some kind, even though the dice technically believe it was something else.
And this is the primary problem. There's a disconnect between what the dice say and the result in play. The die ceases to matter and the results are a certainty. That should be discouraged.
It's not even that uncommon. It's very possible to take a low level monster from full health to dead. Or a commoner. This was an issue with the playtest, where the fighter could never fail to kill a goblin (although the final versions are more robust).
The narrative of the game is that a hit is a hit and a miss is a miss. The language even describes them as such: hit, damage, miss, etc. It's not "you have score a shift" or "succeed at the attack". The language we and the game uses is "hit".
The how of the physical damage is tricky as DoaM doesn't care why you missed. It doesn't care that you're fighting a blinking wizard behind cover who activated shield. There's no dodging involved, no armour to bounce off of, no reason for physical contact. The wizard just, somehow, takes damage out of the aether.
All this would be problematic for a good mechanic. A mechanic with a history in the game and strong place in the system. But DoaM isn't that interesting, and isn't a sacred cow deserving of contorting the ruleset to accomodate.
There's no story behind it. It doesn't add anything to the narrative of the characters, it doesn't sound like a character with DoaM is doing anything special or interesting. It doesn't change the tone of the campaign or allow you to tell stories you would not be able to otherwise.
There are also no real examples of it in fiction, where the hero completely misses and cannot land a blow but still manages to kill their enemy. Or, at least not examples that are meant to be taken seriously. There's no scene in Lord of the Rings where Aaragorn pounds on an Uruk Hai's shield for blow after blow until the creature's heart explodes and it just drops.
The only benefit of DoaM is purely mathematical. It increases your average DPR. And it means you're less likely to "waste" a turn swinging and missing. Which I can understand but is part of the game. No one likes missing, but that's part of the risk. And that's a much bigger issue in a slower paced game. It's less of a pressing need in 5e, with its swift turns.