Unless it isn't (a viable option).
Fights, in ALL versions of D&D with 5e being no exception, are difficult to flee from.
For that not to be the case the DM has to expressly build in a way for the PCs to flee and telegraph accordingly.
And as to the OP, he's stated he WANTS some ideas on how to avoid A TPK because HE made some mistakes. Saying, nah, just kill them - isn't all that helpful!
So as to options:
1. Have an out present itself, such as have an NPC distract the Litch so the PCs can run. Or have something else happen that the PCs see a viable escape route. Make sure the PCs know running is a viable and, likely good option.
2. Have something happen "elsewhere in the dungeon" that requires the Litch's immediate attention, the Litch then dimension doors out to deal with it. The PCs get a breather and a reminder that they need to step up their game or get the heck out of dodge.
3. Have the Lich go full bore, but he needs the PCs alive (for experiments or whatever) and captures them instead. Changes the focus of the adventure campaign but can be it's own fun.
While I'm not against the idea of on the fly modifying the encounter downwards in certain cases. I think, here, there are enough interesting outs to not have to do that.
For the bolded part...
There were rules in 1ed and in OD&D about fleeing.
However...
Most opponents will hesitate to pursue.
Fleeing means using double moves. Which in turns implies that caution is thrown to the wind.
Most intelligent monsters will hesitate to pursue for a few reasons:
1) Are they really fleeing or are they hoping to make me leave my lair so that others will plunder it?
2) Are they leading me into a trap?
3) Am I wounded lightly enough to take the chance to pursue? or could it go the other way around.
At the very least, most opponent might even lose a round or two before pursuing, giving a lead to the group. At which point, it is possible that such a lead makes the opponent to lose track of the character. This is where a good range might be handy to have.
Monsters with animal level intelligence will rarely pursue if lower than 50% of their HP. Their own tendency will be to flee unless in their lair. At which point, they will start their own rest to recover lost hit points...
At the minimum, when characters want to flee. Opponents should make a general wisdom check. If failed, they should not pursue. I do have some guidelines that I use.
Moral Check to pursue or surrender.
Base DC 10
Enemies outnumber the players: DC lowers by 3.
Enemy is a BBEG or have a leader type with them: DC lowered by 3.
Enemy is an evil outsider: DC lowered by 3.
Individual CR of one of the enemies is 5 above the players: DC Lowered by 2
Individual CR of the enemies is lower than the players: DC Raised by 2.
Enemy(ies) is in lair: DC raised by 2.
Enemy is reputed to be fearless or reckless: DC lowered by 3
Enemy is reputed to be coward or have shaky morale: DC raised by 3
- Enemies are wounded below 25% Hit points: DC Raised by 3
- Enemies are wounded below 50% Hit points: DC Raised by 5 (not cumulative)
- Enemies are wounded below 75% Hit points: DC Raised by 7
Enemies are outnumbered by the players: DC Raised by 5.
Enemy is alone: DC raised by 5.
Enemy is a BBEG or have a leader type with them: DC lowered by 3.
PCs have a reputation to slay everything: DC lowered by 5. (applies to intelligent monsters that can have heard about PCs)
PCs have a reputation to be prone to make captive and then release them: DC raised by 3.
*Use the average HP of the creatures.
These are basic guidelines and are akin to the moral checks. I use these and it works quite well.
