I generally dislike the use of passive Perception for secret doors, but this approach sounds good - I will try it out in my next session.
For the most part, I will mix it up. Just like combat encounters, finding secret doors can be easy, moderately challenging, or very difficult. Easy ones...