Going back to the "all or nothing" aspect for a moment.
The question isn't really "does it work all the time to let stuff go vs does it never work?" That's not a particularly useful way to view it. But, we do need to understand the mentality going into things. If it's a 50:50 split, then, well, unless the rewards for letting stuff flee are very, very good, then it's never a good idea to let stuff run away. And, it's unlikely that the rewards for letting stuff run away in a 50:50 system will ever be good enough. If I have to fight the rest of the baddies, losing any chance of surprise 50% of the time and I still have to fight the baddies with a fairly minor benefit (maybe they run away a bit earlier, or they start frightened, whatever), then why would I risk it? Far better to just kill everything and still have a chance of surprising later baddies.
OTOH, if say, the split is a bit more nuanced - 1 in 10 leads to baddies being alerted, 3 in 10 lead to the fleeing baddies just being removed from the field and having no further effect, and 6 in 10 leads to baddies starting frightened until the next long rest (or maybe short rest actually, just to keep the pace of the game up and reward pushing forward), then it makes a lot of sense to start letting baddies flee.
Add in something that 4e did really well - forcing morale failure with intimidation checks - an action to make contested intimidation check vs, say, a wisdom saving throw (I'm spitballing here) - causes a target to be frightened, end of turn save ends the effect. If the intimidation check is made against a frightened target, the target flees the encounter/surrenders if it can't run.
Put these two together and now you have a really rewarding system for the players to use. I might even allow baddies to use this against the PC's, if hte players are okay with it.