While I agree to some extent, usually when the party doesn't recognize a potentially lethal encounter as such, it's DM fail rather than player fail.
Suppose you've got a 6th-level party facing a gang of ten orcs with a one-eyed orc leading them. Those orcs could be 1st-level grunts led by a 3rd-level lieutenant, easily cleaned up with a
fireball from the wizard and a few whacks from the fighter. Or they could be the 10th-level elite bodyguard of a 15th-level high priest of Gruumsh, a TPK in spades. D&D offers no easy way to tell by looking. Even when DMs think they're giving clues about the encounter difficulty, the clues are usually much more ambiguous and less useful than the DM thinks they are.
D&D characters expect to get into fights. It's part of the game. When presented with what looks like a combat situation, against foes who are not clearly superior, it's hardly surprising that the PCs draw swords and attack. Once the fight starts going badly--as you say, most systems make it tough to withdraw from a fight. Somebody will almost certainly get left behind, and no player wants to do that.
As a DM, there
are ways to get the message across, but you have to be unambiguous about it. If my PCs enter a dragon's lair and I want to convey that this dragon is out of their weight class, I don't portray it as Large or Huge. I grab
Big Red and bang him down on the table. If they meet an orc warband and I want them to realize a head-on confrontation isn't going to work, the warband isn't going to have twenty orcs in it--it'll have two hundred. Or, after giving them a tough battle against a single monster of a given type, I'll have them run into nine or ten of the same critter.
If, in
these situations, they still pick a fight, I'll serve up a TPK sandwich with all the trimmings. But if they get into a fight with a gang of orcs that just happen to be a few levels higher than they expected, I'm not going to claim that they should have somehow known better. And if for some insane reason I have an overwhelmingly superior foe initiate hostilities, I'm not going to blame the players if they stand their ground and die like heroes instead of running away.
IMO, this is so right that it should be the opening chapter of the
RPGers' Bible.
Not being able to tell if the opponents are useless minions trained at the Imperial Stormtrooper Marksmanship Academy or Elite Supersoldiers with laser designated targeting is not unique to D&D, pretty much every game I've ever played has its degrees of ambiguity. And most gamers expect to have to fight their way out of bad situations sooner or later - whether they're playing D&D adventurers, Edgerunners or questionable "merchants" plying the "space lanes".
There should be no ambiguity about the situation, especially if the party contains at least one member who is going to have every reason to fight unless such an action would clearly be suicide.
It's up to the GM - who is, after all, the only one there who really knows what the world looks like and is charged with the responsibility of conveying that to the players - to convey to the players that the situation is inescapable and that resistance is, indeed, futile - if not downright suicidal.
The players do need to keep their wits about them and listen to the GM, but they shouldn't have to hunt for subtle clues ("but you should'a' known by the fact that they were wearing different-coloured shirts that they weren't going to be the same pushovers as you met before...")
Having one of the opponents articulate "If you come quietly, I promise you will not be harmed but we will kill you if you resist" probably would not cut it for some PCs - it still boils down to "Surrender or Die!" and if "surrendering" is going to mean dying later in chains at the hands of torturer/executioner for one of the characters, that's not gonna happen if (s)he thinks there's the remotest chance of escape - "there's only a few of them, we can take 'em out and escape..."
The only answer, IMO, is to ensure that the players can see that initiating hostilities will quite literally be the last thing their character ever does...
If they then elect to go out in a blaze of glory, it's TPK with da woiks and extra pepperoni - and
anchovies, whether you bloody like 'em or not.
And unless the party does something that warrants it, I would not have a clearly superior force initiate hostilities - and I suspect that if the players did do something that warranted a death squad, they'd probably feel they have no reason to expect a "fair trial" and good treatment and initiate hostilities anyway - "at least go out fighting..."
When I'm playing, my expectation is that I should be able to assess the level of risk and react accordingly, based on what my character - whose background motivations and such are
known to the GM - would realistically do under those circumstances.
That would include resisting being arrested by a smallish group of soldiers working for an inquisitorial organisation that's rounding up and slaughtering the likes of me.
In the case of the OP, if it crashed the game due to the GM being so inexperienced that (s)he didn't even create stats for the opposition, I'd say "cool, well let's call a break to give you time to come up with some basic stats, I'm heading outside for a coffee and a cigarette, we can get back into the action after that and see if they succeed in taking us."
Hell, I'd even be inclined to yak with the GM while I'm having my break and give him/her some ideas for the characters or how to retrieve the situation - but I'd still have reacted as per the OP and fought rather than submit to capture, torture and execution.