Ashrem Bayle said:
In reality, the greatest swardsman in the world will die from a blade through the heart, even if a peasent girl is the one that puts it there. D&D cannot handle this.
D&D =/= Reality
You can interpret the dice rolls and mechanics in many ways, but is it really that awful that your party's rogue didn't one-shot a random enemy?
DM: "The warrior of the Red Hand army chuckles to himself as he looks out the window, staring at a cloud formation that vaguely resembles something provocative--if you squint at it just right. You raise your blade, focusing on the small gap between the scales of his armor above his left kidney, and tense your muscles in anticipation of the attack. Roll to hit!"
<dice roll>
DM: "The subtle movement of the air behind him causes the warrior to shift on his feet, just as your blade strikes home. This warrior must have seen more battles than you anticipated, and contorts his body to absorb some of your thrust, turning what would have been a lethal blow into gaping flesh wound instead. Wincing in pain, he turns to face you, with his mace held high. Roll for initiative."
Player: "What do you mean he didn't die, I totally sneak attacked him!"
[Player2: "Where are the Cheetos?!"]
DM: <sigh> "Well, his fighting instincts seemed to kick in, and he was able to survive your attack. You should probably roll for initiative now..."
Really, isn't this exactly the kind of thing HP is supposed to abstract? How does a 5th level fighter survive a critical hit from a battle axe when any first level character won't? Once again, maybe the enemy wasn't meant to be the "mook" you thought he was, if he had that many HP. Maybe the encounter wasn't meant to be a pushover. You seem to be blaming the DM for not changing the scenario to match your expectations, and not considering the idea that your expectations were wrong.
A "realistic" game would mean that any single hit could be instantly lethal to anybody, and plenty more hits would be mortal wounds that you can't recover from. While such a game could conceivably be enjoyable, I like to take my reality in small doses when roleplaying, and not being able to one-shot any given enemy is something I'm willing to accept if it makes the game more fun. YMMV...(and obviously does!)
Even with all of it's abstractions, D&D contains many "save-or-die" effects, including death from massive damage. If your rouge wasn't an Assassin PrC, and couldn't deal 50+ HP of damage on a single attack, why should he even expect a one-hit kill? If that's what the player was after, he should have asked the DM if his attack would be considered a coup de grace (which the DM could rule in favor of, even if the enemy wasn't helpless), since that includes a save-or-die effect based on the damage dealt. But if the rogue didn't think to ask that up front (or did and had the idea rejected), why should he be upset afterwards when his assumption that the enemy had low HP didn't pan out?
I think at the most fundamental level, this is more a disagreement between the players and the DM. It could happen in any roleplaying system. The players had a certain expectation that was different than the DM's for a given scenario. It's bound to happen in every campaign, regardless of the system's rules... I don't think it's fair to blame the disagreement on D&D being any more or less "realistic" than any other roleplaying game, because they all have some level of abstraction. And if the players don't enjoy the game because they don't see eye-to-eye with the DM on the way the game is supposed to play out, then changing your abstracted rule system isn't going to fix the problem.