I do the "eyeball it and make a Wisdom save or they book it" method, advantage/disadvantage if circumstances are relevant- but Kobold Press' excellent GMG has a more in-depth version in its optional rules, if you're into that:
MORALE
Some creatures lose the will to fight when they find
themselves outmatched or outnumbered. This optional
rule can help you determine if a monster would flee in such
a situation. When using this rule, first determine if the
situation might test a creature’s morale and cause it to flee
and then have the creature make a WIS save to determine
if it actually flees. If the situation doesn’t change or if it
worsens, the creature might have to make the save again
the next turn, even if it succeeds on the first save. Morale
should only ever apply to monsters, not PCs.
An individual creature in combat, or a creature that isn’t
fighting alongside a partner or group in larger combats,
might have its morale tested and flee under one of the
following circumstances:
• It is reduced to half its hit points or fewer for
the first time in combat.
• It is surprised.
• It starts its turn and is aware that it has no way
to damage any of its opponents.
A group of creatures bolsters the resolve of individual
creatures, but the group’s resolve quickly crumbles if its
leader flees. A group’s leader, or even the entire group,
might have its morale tested and flee under one of the
following circumstances:
• A member of the group starts its turn and is aware the
group has been reduced to half its original number or
fewer while its opponents haven’t suffered similar losses.
• All the group’s creatures are surprised.
• The group’s leader is reduced to 0 HP, incapacitated, or
otherwise removed from battle.
When a situation might call for a creature to flee due to low morale, that creature must succeed on a WIS save or flee (DC equal to 10 + the creature’s WIS modifier). A creature has advantage on this save if its leader is still alive and visible, if it knows allies are on the way, or if it has some other knowledge of the situation that might help it overcome its current fears. A creature has disadvantage on this save if its leader has been slain, if its other allies have already fled, or if it has some other knowledge of the situation that might heighten its fears. A creature immune to the frightened condition automatically succeeds on this save, and a creature resistant to the frightened condition has advantage on it.
Fleeing Creatures. A creature that fails its morale check flees from combat. A fleeing creature must take the Dash action and move away from combat by the safest available route on each of its turns. If there is nowhere to move or
no clear exit to the combat area, the creature moves as far away as it can from the last creature that dealt damage to it (or the creature that felled its leader or other ally if it hasn’t taken damage) and then cowers in place, taking the Dodge action if able. The creature surrenders if combat otherwise ends, but it protects itself and resumes combat if met with hostility or aggression.
Rallying Allies. A leader of a group can take a bonus action to rally a fleeing ally it can see within 60 feet of it by succeeding on a CHA check (DC equal to 10 + the target’s CHA modifier). The rallied ally is no longer fleeing and can rejoin combat. A rallied creature might flee again if that leader is slain or flees or if another situation tests its morale.