Narratively, hit points are the devil. You can get crit for half your hit point total, but mechanically you're fine. Sure, we can explain this to a degree as your will to live and adrenaline counters any "penalty", but that only goes so far. You can try to explain that hit points aren't wholly physical in nature, but then the question of "when do I take real damage" rears it's ugly head- Micah Sweet likes to bring up secondary effects, like a snake bite inflicting poison, and it's a fair cop. If you didn't actually get bit by the snake, how did it poison you?
On the other hand, games that try to model the actual effects of wounds can be unfun to play, unless the goal is to avoid situations like combat entirely (which is valid for some games, but not others). This quickly leads to the "Death Spiral", where as you take more damage, the odds of you taking more damage increases. Shadowrun, Vampire: the Masquerade, the original Legend of the Five Rings game, Star Wars SAGA, and many more do this.
Alternately, you have systems that have hit points, but delineate which hit points are "real", generally a small pool, and "temporary". Star Wars d20 had to be my worst experience with this. Some attacks ignored your "vitality" and went straight to wounding you- like lightsabers. You know, the thing every starting Jedi is going to have. Critical hits in Fantasy Craft have a similar effect, and there's something disheartening about going from "I'm fine" to "I'm dead" due to random chance (and a GM who has been stockpiling action dice!).
Permanent effects are the devil, especially if the game has no easy way to recover from them. AD&D (both editions) made it very clear in the DMG's how sorry your characters would be by about 5th level or so, missing hands, limbs, eyes, ears, and the like. Since the game was about fighting monsters and taking their stuff (1e gave you reasons to try to avoid combat, but 2e did not), this wasn't ideal.
I first encountered permanent wounds with games like RuneQuest. A friend of mine was trying to sell us on playing the system. He recounted a tale about how, with a mighty blow, his character once attacked a foe, hit them in the arm, penetrated their armor, did enough damage to sever the arm clean off, then continued to inflict damage to the enemy's torso.
"And that can happen to players too?"
"Of course! Isn't that fun!?"
A permanent injury of that severity can, even if you survive, force you to retire a character. There's certainly a time and place for high PC turnover in games- Paranoia is built on this, with readily available clones to replace your characters as they die in ridiculous ways.
But if your game gains value from characters sticking around for awhile, raising the stakes to the point that actually dying is a tragedy, then that's not so hot. This is why I stick with hit points, even with all their warts and flaws- it may be unrealistic, but it suits the kind of games I enjoy playing and running better. Often, you're knocked down in a fight with no real agency- you were in a fight, enemy roll good, you lose hit points. Some people think that you should suffer a penalty for going down, but I've found that to just be unduly punishing- your prize for being in combat is penalties that make you not want to be in combat. Yay.
Some games (I have dim memories of Albedo) also have mechanics for shock and stun due to being in a firefight, where taking damage might force a Coolness Under Fire check, and failing that causes your character to skip turns of combat! That's certainly realistic, but boy, can it lead to players "checking out" of the game, quick!
Again, if the object of the game is to avoid combat: V:tM (though you can't seem to convince the players of this fact...), Call of Cthulhu, GURPS (you can't convince me otherwise!)- hit points and no penalties for damage is probably not the way to go. People do tend to get laissez-faire about their character's fate if they have 99 hit points, after all!
If the game is about fighting giant, fire breathing dragons, and surviving to fight more, well, limited hit points, penalties, and specific injuries might not be.
There can be middle ground, however! FASA's Earthdawn gives you a pool of "hit points", but taking damage above a certain threshold could inflict a wound. Removing a wound took one of your Recovery Tests (think Healing Surges, back in the 90's!). Alternately, you could struggle with a wound, but each wound gave you a stacking penalty on attempts to heal damage, until you were simply forced to take the time to remove them. It's a system I've often toyed with applying to 5e, but I have to figure out how to con...vince my players into buying into it, lol.