First, I telegraph in some fashion that secret doors are a thing in the adventure location. That might be mentioned in the exposition somehow ("The Blue Knight used a series of secret doors in the keep during the battle to surprise the goblin invaders in several skirmishes, but ultimately it wasn't enough...") or there might be a secret door that is already opened somewhere. Basically I try to say, via describing the environment, that you might find secret doors around here if you look for them.
Then, I get each player to tell me what the character are generally doing while exploring the adventure location with the understanding that unless you're a ranger in favored terrain, you're generally either Keeping Watch for Danger or doing something else distracting from that (some exceptions may apply). They can, for example, Forage, Navigate, Draw a Map, Track, or do something like Search for Secret Doors. While doing these tasks, they are at risk of being automatically surprised if Team Monster is lurking. (But not all monsters try to gain surprise, so it's not too bad a trade-off.) Players can, of course, switch their task to something else at will.
If someone chooses Search for Secret Doors as his or her general task, then passive Perception applies while the character travels the dungeon. If that score meets or exceeds the DC for the secret doors the party passes along the way, the character is rewarded for the risk he or she has taken by noticing faint cracks in the wall or scuff marks on the floor, something like that. Then that character or the rest of the party can investigate to figure out the mechanism by which it opens which may call for an Intelligence (Investigation) check if there's an uncertain chance of figuring it out and a meaningful consequence of failure. If they can't figure it out, they can break it down, but naturally this takes time and makes noise which may call for a wandering monster check.
To me, secret doors are almost always a bonus for the PCs. They either lead to treasure, a shortcut, valuable (but not required) information, or some other resource that is nice to have, but that they don't necessarily need (such as a safe space to rest in the dungeon or whatever). If nobody bothers to choose to Search for Secret Doors, the adventure won't be stymied as a result - they'll just be potentially passing up some goodies.