I entirely agree. The view of hit points that I presented is hard to align with events which should be directly damaging.
For example, there is little that can be done to avoid damage from a fall. Then, a more realistic way of handling fall damage might be to represent it as a coup-de-grace. Or to use escalating damage (1d6, 3d6, 6d6, 10d6, and so forth).
But, most games don't use these more realistic methods. My sense of this is that, having accepted hit points as a measure of survivability, folks prefer to dial back the lethality of events such as falling. (Which, in my experience, matches an avoidance to use of coup-de-grace rules. They are there, in 3E, but they don't seem to be used much.)