Exactly my point, people should be able to successfully stealth against a group like that, and that is handled by passive Perception.
Similarly, and the bring this thread full circle back to the point of my original post, this seems to work well similarly for traps, use the PCs' passive...