The worse is when the party retreats, one person gets caught, and the entire party comes back, which is enormously counter productive....
"Oh, '
fly you fools' - we thought you said 'follow, fools!'"
Dead characters learn no lessons.
But players do, thus is meta-gaming (or was that 'player skill?') born...
I think it is a mistake to design an encounter in such a way that fleeing is the only option. I don't think encounters should be scripted like that. If you set up an encounter where the players must flee the dragon, they WILL fight it. I guarantee it.
It's one of those things traditional D&D play styles have issues with. D&D was something of a puzzle-solving, treasure-hunting variation on a wargame in the past, one puzzle was whether to fight or flee, that is, to choose your battles. Fairly quickly, though, people started treating it as an RPG, instead - an heroic-fantasy RPG, even, sometimes, and there's a world of difference between 'heroic fantasy' and 'choose your battles wisely.'
So there's this impulse to pull in dramatic genre tropes, but no support from the system for it, and, even where there is support, a culture of play that doesn't expect or avail itself of it. For instance, one issue has generally been that a monster calibrated to challenge a very high-level party is not something a low-level party can hide from using opposed checks. Not a chance, really. Like the dragon has +32 spot and a +35 listen, and the party stealth scores range from -3 to +17... oh yeah, that'll work out. They can know they shouldn't fight it all they want, they're not hiding from it, they can't outrun it. What? trick it, negotiate? Go for it, sense motive is +33.
CR and Threat Level and all these concepts were introduced by D&D, and many people who have only played D&D seem to think that all encounters must be balanced.
A 'balanced' encounter is just one that's going to (if it's balanced correctly, rarely the case in D&D) present a challenge appropriate to it's level. A Dragon might be a balanced 19th level challenge, it still is, when a 9th level party encoutners it. Still balanced. You're dead, but it's still a perfectly balanced encounter. Too bad you were 10 levels lower than it...
Dude. You're the only one who's even read the DMG!
...but, I already know how to DM... why would I read it?
"Run away! Run away!" ... what if they don't?
TPK. If that's not a bad enough result to teach them to be better at running, make them sit through Monty Python and the Holy Grail after each TPK.
Now this is not a universal opinion, but some DMs (and I'm one of them) believe that not all encounters should be balanced. Once in a while your party should encounter bandits and crush them, but also once in a while the party should encounter a monster that is just *too much*. The encounter isn't a "fight the monster!" challenge, it's a Let's get the hell outta here!" challenge. If the party manages to hide, move quickly, create a distraction etc etc, they can escape.
BUT... what if they don't?
They get squashed if that's the stakes you set. Or, maybe they get captured & escape later, or captured and bargain their way out by promising to perform a quest for their captors, or whatever... or maybe they get killed, wake up in Hell (even though none of them are evil nor sold their souls, and the adventure continues from there...)
It's the GM's job to telegraph that the incoming monster is *particularly* dangerous. ..It should be really clear that this monster is exceedingly dangerous, and I think most GMs are up to the task. It should also be an "escapable" monster
This is where a formal sub-system, like stakes-setting in an indie game, or even just a skill challenge, can be helpful, it frames the challenge and gives the players a sense of whether it can be overcome.
So... what to do if this happens? I think that "oh, look at that, those Fire Giants all had 14 hit points each, lucky you!" isn't a great solution for the GM. Anything else that isn't lame?
TPK & capture scenarios are the obvious first steps. (I don't care for it, but you could awlays have some Deus Ex Machina swoop in with some giant eagles and fly them to safety.) From a TPK you can build a new party, or have the old one raised for some weird reason (possibly Ressurected or miss-wished back to life at a much later date), or follow them into the afterlife with a chance at getting back. Capture scenarios are familiar from genre, but fraught in D&D for many reasons (you have to beat the party to 0 hps to capture them, and nullify casting to hold them, so they won't be up to fight their way out for a while, many D&D characters are extremely equipment-dependent, so loss of gear can be worse than death, etc).