Yeah...
Look at it this way: If Evander Holyfield gave you a roundhouse to the jaw, would you be in serious trouble? I know I would. I would probably end up with a broken jaw and a concussion - at least.
Now, if Evander Holyfield punched Mike Tyson in the jaw, would he shrug it off? No tonly would he shrug it off, but he would then bite Holyfied's ear off.
So, in conclusion, a punch that would floor your average Joe (equal to a 1st level Commoner) would only make Mike Tyson (probably a high-level Fighter) angry.
Some might say that with weapons it would be different. I dont think so.
Ok, lets compare Joe Average(who's never been stuck with anything bigger than a sewing needle) with a toughened brawler who's been stabbed a dozen times.
Joe gets stuck in the arm. He immediately passes out from the pain and shock. Maybe he dies of blood loss while he's unconscious.
Brawler gets stuck in the arm. He has been stabbed before and knows what it feels like. He does not pass out from pain and shock. He's used to the pain and the blood, so he just grits his teeth, yanks the knife out of his arm, and rips into his assailant.
Once again, Joe is a 1st level Commoner, the brawler is a high-level fighter.
Conclusion: What doesn't kill you, makes you stronger. Someone who has not experienced a lot will not be as tough as someone who has.
That's my take on HPs