I probably wouldn't have ever used passive perception at all if someone hadn't taken Observant at a table I was running for. To me, Perception should be invoked any time there is something to notice, it should never require an action- could you imagine if you walked into your apartment and died because you didn't take an action to smell natural gas buildup due to a pilot light going out?
I liked the idea of passive perception when I ran 4e, but in reality, saying "oh sorry mate, you missed detecting that guy sneaking up on you because your passive perception is 12 and he rolled a 13 on Stealth..." no, Perception and Stealth should always be opposed rolls, there shouldn't be a "take an action to notice a guy sneaking up on you". Or trying to pick your pocket, for that matter.
And you can have bonuses to die rolls like guidance or even a Bardic Inspiration that don't apply to a passive check, because they aren't die rolls*. It's all kind of messy and murky. If I don't want players to "know", I can just roll behind a screen for them if I want to, but honestly, I always roll openly, and I don't care if the players know what they rolled.
If they want to act cautious after seeing a low roll for no reason, hey that's fine- I have a tendency to use "fake rolls" from time to time anyways, a trick I picked up from one of my first DM's.
*There is the argument that you shouldn't be able to use these because you don't know you'd need the bonus, but the game has plenty of bonuses a player can apply to things you probably shouldn't be aware of, and if you fall down this rabbit hole, then justifying spending a Bardic die on a saving throw or casting shield (or heaven forbid, silvery barbs starts to become suspect, let alone using Inspiration to gain advantage).