Right before the fight a DM can make a morale check to see how willing to fight are the monsters. On a fail then the DM use any event he judge appropriate to make monsters retreat, flee or surender.
The same way DM can use a tactical check (int) to to make monsters more or less efficient or...