Sure, but why get rid of the fun stuff?
Honestly, 90% of secret doors add nothing to the game except frustration. Do they sometimes make sense from a narrative standpoint? Sure. Do they create interesting play scenarios? Almost never.
They’re an artifact from early editions of the game which were focused on dungeon delving and involve exploration turns and wandering monster checks and resource management.
Removed from that mode of play, all they really appear to be is a roadblock for progression gated behind a skill check. Or in the case of some GMs, behind arbitrarily specific action declarations like “I run my hand along the underside of the fireplace mantle”.
If the game isn’t focused on delving, then the interesting stuff is likely whatever’s behind the door, not just finding the door. Skip that nonsense and get on with it, I say.
There’s already far too much “not quite yet” in D&D. No reason to find additional spots to insert more.