Yeah, the rules and general consensus make it seem rather conclusive. They get a chance to perhaps notice the secret door if they make their Search check. Just because they know the door is there, doesn't mean they can get through it.
My first reaction when reading that was "It's totally unrealistic, though." Then I realized that I'm playing D&D and nothing is 'realistic'. I might talk that over with my PCs; thanks a bunch!