Out of curiosity, did you do the whole "The ogre swings his club and misses; you take 17 damage!" thing? And how did you come to that interpretation, based on the rules in the book (if you remember)?
I can't know the specifics of your history, or the histories of everyone in your gaming group, but more data is always useful.
No, it was never simply described as "The ogre swings his club and misses; you take 17 damage!" I always try to make combat sound more interesting than that.
The narrative explanation for non-physical HP loss generally went more like these examples:
1) Deflection example: "The bandit thrusts his sword at you, the tip racing toward a weak point in your armor. At the last second, and with great effort, you deflect the blade. You take X damage."
2) Surprise attack dodge example: "You suddenly hear the straining of leather from behind you. Turning to face the sound, you stagger back a step as the assassin's poison dagger narrowly misses, putting a small scratch in the chest of your leather armor. You take Y damage." If you remove the surprise attack element, this also counts as a clothing/equipment damage example.
3) Luck example: "The cultist raises his dagger, catching you momentarily unprepared. Expecting the blade to slam down and pierce your flesh, you sigh in relief as the cultist is bumped by one of his adjacent allies and has to draw his hood out of his face. You take Z damage."
It's not as if we exclusively used non-physical HP loss. We liked having the narrative freedom that both options allowed. Here are some examples of HP loss that were mostly non-physical but had physical elements:
1) "The mercenary slashes his scimitar at your face. You lean back, barely evading the blow, your hair trailing behind and being neatly trimmed by the gnarled but still lethally sharp blade. You take X damage."
2) "The ogre's club comes crashing down toward you. You barely get your shield up in time, but the impact is much stronger than you were prepared for. Your muscles strain to keep the blow from tearing the shield out of your grasp. You take Y damage."
3) "The arrow darts toward your face, the tip glinting with the light of the torches in the room, and you barely avoid losing an eye as the bladed head of the arrow lightly grazes your cheek. You take Z damage."
Of course, I also implemented an injury system (although this example didn't occur until 4e, whose disease tracks I used as the basis for my injury rules).
"You find the floor is suddenly no longer there. You experience an odd sensation of semi-weightless just before plunging into the pit. You take X damage, which is in excess of your injury threshold. Roll to avoid injury (roll fails). You land feet first and crumple into a heap with a loud crunch that, as you attempt to move, you suddenly realize is a broken leg."