If they set things up, then you design the encounter to have favorable man-made terrain to make their initial stealth checks with. Start the sneakier guys -behind cover- where Stealth checks don't mean a damn, as they start off unseen anyways.
I'm guesstimating your Cleric is either a level 14 cleric who started with 18 wisdom and took Skill Training/multiclassed to get perception, or is a level 24 cleric.
1) If you want to use stealth against your party, use stealth monsters. Night hag (level 14 Lurker) has a Stealth of +16, which your cleric is still good against, but it isn't auto lose. If it's the level 24 variant, taking 20 means your monster is never seen, period, with +24 to his checks.
2) If it's the former case, then the Cleric spent a feat or two to be perceptive. Instead of changing the rules just so you can continue to ambush the party the old 'cause the dice said so' way, create encounters where you flat out say 'Alright, here's the monsters you see' while having monsters use terrain in a way the Cleric -cannot- see them until the last. This way, the Cleric can feel like a hero for seeing monsters who hide (which is fair, he spent valuable character resources into being able to do so), and players who are NOT the Cleric can actually have a chance of seeing monsters -at all.-
It's not an arms race, this D&D.