When dealing with encounters, when they start and when they end, it's important that the DM and the players get on the same page from the beginning. If the players trust the DM then the DM can have some leeway in it. If they don't you could set a time limit; one minute after the last combat roll was made, the encounter ends.
I also don't think players should say "I try to regain my abilities", especially not as a way of seeing if enemies are close. If the enemies are close but passive, then I don't see that as an encounter.
My take on the two situations (the displacer beast and the invisible mastermind) is as follows:
Displacer beast: The bandits are slain, the heroes have won the battle, and when the last bandit falls the displacer beast charges in. As the DM you can't be surprised by a situation like that, the displacer beast is there because you want it to be and therefore you should know from the beginning if you meant for it to be a new encounter or a continuation of the old. If the players argue differently, see the beginning of the post about getting on the same page as the players. If a player still argues, then explain the meta-game reason; "I planned for you not to have your per encounter abilities since I think the displacer beast would be too easy otherwise and I wanted you to feel under pressure." If he still doesn't buy it, per encounter abilities aren't your problem as much as a player that doesn't agree with how you run the game.
Invisible mastermind: Again, you must have a plan with his behaviour. If he, for some strange reason, wait until the last of his minions fall before breaking invisibility and attacks, he has made a crappy tactical decision but still acts in the same combat. If he, OTOH, hangs around for a good opportunity to strike then I would consider the encounter to be over.