Because of a new campaign I'm starting, I've been clarifying some of my house rules and standard rulings. One of the rules is how I handle passive skills, such as perception and it might be useful here.
The way I usually handle things is that passive skills will usually give you general information, but not something specific. So for secret doors, you might notice that the room takes up less space than you'd expect based on adjacent rooms. Or that the bookcases on one of the walls doesn't look like a single unit as the other walls. The fireplace is oddly off-center, or the soot pattern is strange. That sort of thing.
Then they would describe what they are doing, in combination with a skill check assuming they don't just figure it out on their own. I use the skill check in this case to fill in a blank where character knowledge might be greater than player knowledge. I also might modify the skill check based on their actions. I usually don't penalize the check, but I might give advantage.
There was a very funny situation in my campaign where a chest had a false bottom. They player investigating it described practically everything he could think of (is there an extra compartment under the lid? I examine the outside of the chest, etc.) except to examine the bottom of the inside of the chest. So I had him make a skill check. Several, actually, and they all failed miserably.
They decided that there was something important about the chest, so they put it on their Tenser's floating disk and took it with them. When they came to a long staircase they threw it down the stairs and finally figured out what they were missing when it broke open!
Ilbranteloth