Yeah, it's actually two separate issues though: (1) artificial constraints imposed by initiative & (2) D&D 5e not having rules for escaping/withdrawing from combat.
I use a whole variety of house rules, which may or may not suit your table/tastes, but these are some of them...
Dynamic Initiative
Dynamic initiative acknowledges that there are degrees of combat scenes, and they are served by different approaches to initiative.
Snap Scenes: When there’s just a few monsters of the same type, like a quick guard scene, it can be resolved with the active/leading player rolling initiative against a DC of 10 + the monster’s DEX. PC wins? They go first. PC loses? Monsters go first. Sometimes, the combat is so secondary to the scene that it can be resolved simply as “resolve the PC’s hostile action.”
Typical Scenes: For most combat encounters, initiative is not rolled and the round begins with whoever triggered the scene (if in doubt, have one PC roll versus one monster to determine who goes first). When that character finishes their turn, they choose the next creature/group to act, and so on. The last person to act in the current round decides who starts the new round – but they can't pick themselves.
A creature/PC that hasn’t taken a turn yet this round may interrupt the order if it took damage (or if it spends Inspiration or a Legendary Resistance).
Climax Scenes: For climactic / set-piece / boss encounters, everyone rolls initiative. If the PCs are not surprised, the players may have one minute to make their plan of attack. During initiative, players who have consecutive turns with no monsters in between them may act in any order they wish, including overlapping their turns.
Retreat
When the party chooses to retreat from danger, at any point on a player’s turn they make a group initiative check versus the passive initiative of their foes. A PC who is unable to move does not roll. A PC who is encumbered, carrying another PC, or slowed suffers disadvantage on the roll. A PC who had an escape plan in advance or who took action during the scene to facilitate escape gains advantage. Same goes for the monsters/NPCs.
If half or more of the PCs succeed, they escape. Otherwise, the scene continues and escape is no longer an option unless circumstances change. However, if they escape, for each PC who failed the check, the party must pick one:
- The party has been split up, and possibly lost.
- The party abandons the treasure or loses a significant item.
- Each PC takes damage equal to a single opportunity attack from the monsters/NPCs.
- Even if the monsters/NPCs could not, or chose not to, immediately pursue, they’re out there looking for the party.
- The party lands in some new sort of trouble.