Long rests SHOULD fix everything. HPs are about being winded and minor scratches. Lost HPs should disappear after resting. If we want a permanent/persistent damage mechanic, that's a different issue. Do you have an alternative for that?
So your interpretation is that it's like one of those anime shows where the characters just get progressively more scuffed up but nobody actually draws blood even if someone gets thrown through a wall and lands on a bunch of jagged rocks?