Yes, but HP isn't solely a reflection of one's ability to withstand physical damage - it's a throwback to the old days where it was a reflection of luck, favor of the gods, ability to avoid damage, etc.
I have no problem with it precisely because HP =/= Physical.
HP = Physical + Spirtual +...