I completely agree. Simulationism and armor get really weird in D&D, especially since there are so many different things that fall under the "armor class" umbrella, and so many ways to define "damage." Everyone finds their own way to square this with the fiction of their game; I'm no exception. My brain parses it like this:
In my mind, a "miss" is an attack that fails to do damage (or fails to remove hit points, if you don't like using the D-word.) A sword can fail to do damage/remove hit points in a number of ways: it can fail to make contact with its target, it can glance off of armor, it can get parried aside by an opponent's blade, it can strike a shield or magical force field, whatever. The same goes for daggers, axes, spears, clubs, arrows, maces...and yes, hammers. (Tangent: why hammers?)
If the attack deals damage in any way (or removes any amount of hit points), I don't call it a "miss." Instead, I measure the amount by making a damage roll, which will inform everyone if the attack was a devastating blow (rolling max damage), just a flesh wound (rolling minimum damage), or anything in between.
I'm not saying this is the right way to parse weapon combat in D&D, and it's certainly not the only way. It's just the way that I will always use.