I like the idea of a Wisdom check instead of a Will Save. There are several reasons for this.
First, it is much simpler to scale it with level. For CR 2 monsters, the wisdom score is generally 10-16 (about +2 on average). For CR 15, it is 15-24 (about +5). A static (or nearly static) DC should work well. In constrast, Will Saves have gone from 0-5 to 13-18, a net change of about 13.
I also think that a Wisdom check is the appropriate check. Wisdom represents common sense and your ability to apply knowledge. An Int check might apply to notice that the battle is lost, but wisdom applies to whether you do the wise thing (surrender or run away) or the foolish thing (stay and fight).
Combining several ideas from other people, the most complicated system I would propose is this. In all cases, sizable circumstance bonus (probably up to about +/- 6) could be applied based on the personality and constraints on the monster/NPC.
Start of battle: (optional) DC 20 Wisdom check to decide not to fight.
Battle is going poorly:
- (Optional) DC 15 Int check to realize battle is going poorly
- (Optional) DC 15 Wisdom check to decide to flee/surrender
Battle is hopeless:
- (Optional) DC 5 Int check to realize battle is hopeless
- DC 10 Wisdom check to decide to flee/surrender
The actual numbers have not be tested at all, and are a plausible guess. They can certainly be increased/decreased based on your personal preference of how often foes should run.
The idea behind the int check is that intelligence is about gaining information. If you do not know that the battle is hopeless, it will not matter how wise you are.
"Going poorly" and "hopeless" are probably need to be based on GM judgement. I can see rules like "50% hit points" or "party at 30% of total hit points" but you would also want to consider status effects. If the PC's are hasted and have short term buffs giving them +5 to all rolls, while the monsters have -10 to all rolls, I think the tide has turned even if everyone is at full health.
