Sure, my issue is purely with how much it gets rolled, and when it gets rolled.However, given that Perception is a skill in the game, it's understandable that people are going to want to roll for it.
Agree, but there are simple ways around this. First off, I would not propose never rolling, but I'd propose only rolling for things where it's sort of dramatic (ambushes, a briefly seen object, an attempt at pickpocketing), and not merely about "did you hear at the door" or "did you notice the secret door"? I will note that, inconsistently, some WotC material does use passive Perception that way, but not most of the time.Having purely passive perception could be seen as less fun - either you automatically find it, or you automatically miss it. and since the DM setting the difficulty also knows what the passive scores are, the DM is basically deciding in advance if the party succeeds or fails.
I'd also personally have made it so that basically anything physical you can notice with Perception as a passive DC you can find with intentional Investigation with a significantly lower DC. I think it's really wack that passive Investigation even exists, btw. Investigation is, by definition, an active task, and one you can definitely screw up in ways you can't really with Perception, even if you attempt to be methodical (which can itself be a mistake, or at least waste time). I've personally found very few situations where it genuinely makes sense to use passive Investigation - they're significantly rarer than active Perception checks. Passive Investigation feels like one of the various "half-baked" rules concepts that doesn't actually see much use, RAW/RAI, only if an enterprising DM decides to make use of it somehow.
Continuing I think that making it so social cues and spotting if someone is lying and the like come off Perception is a bit silly, as it's such a poor model for fiction and so irrational. You simply cannot (I would argue) be all that good at Deception, Persuasion, or Intimidation without a strong ability to read social cues, and to understand when others are doing the same. And there are plenty of people who are amazing at reading social clues who would be extremely easy to sneak up on. Right now we have a funny but stupid situation where one of the PCs (with high CHA) does all the social stuff and another acts as the lie detector (with high WIS). It's not like that's an unheard-of dynamic in fiction, but it is pretty silly-feeling, imo.