If you're going to resolve finding secret doors with passive Perception while the PCs are traveling the dungeon, then I suggest you make that a trade-off against keeping watch for danger. In other words, you can't do both, unless you're a ranger in favored terrain. Therefore, the character has a chance to notice secret doors, but is automatically surprised if a stealthy monster comes calling.
Rather than do it that way though, you can also do it the way
@Charlaquin suggests and telegraph the existence of the secret door in some fashion and then, if the PCs search the room, whoever is specifically searching for secret doors in a way that speaks to what was telegraphed can either find it outright or you can call for an ability check. In most dungeon environments that I run, I break down basic dungeon tasks into 10 minute blocks of time and then either have wandering monster checks every 10 minutes or every hour. This builds in a meaningful consequence for failure for these tasks.
So, for example, the PCs enter a chamber that the DM describes. The fighter keeps watch, the cleric searches for traps, the rogue looks around for stuff to loot, and the wizard searches for secret doors. This all takes 10 minutes. Resolve each task in the order that makes the most sense, asking for ability checks when necessary (Wisdom (Perception) for secret doors), make that wandering monster check (or tick 10 minutes off till the next one), and throw a monster at them if called for. If that monster is the stealthy sort, everyone but the fighter is surprised automatically. Once that's done, describe the situation as it now stands and ask what they do. Repeat till they move on.
Also remember that just finding a secret door doesn't have to be the beginning and end of the it. Sometimes figuring out how it works is also complicated and, depending on how the PCs go about it, it may call for an Intelligence (Investigation) check to make a deduction about how it operates. This is another 10-minute task in the above framework which comes with further risk of wandering monsters (or moves the clock closer to them or closer to some other time limit).