A large part of our disagreement is going to hinge on a subject that I have little interest in debating: the level of abstraction used to interpret what hit points mean. I'm fine with a high level of abstraction. I suspect you aren't. So to me, it makes sense that a seasoned fighter can swing and not quite hit you, but it still wears you down to avoid his well-placed strikes, so you become fatigued. How do we model that fatigue? A loss of hit points.
Now if you do disagree with me about abstracting hp mechanics to such an extent, you aren't wrong. We just have different tastes. It will lead you to dislike mechanics like the slayer's damage-on-a-miss while I admire them.
At this point it's mostly a matter of taste, so I think we can just agree to disagree.