I don't get that impression.
The section you quote is in the part of the book talking about exploration turns and the actions you can take during exploration. That section also says that exploration turns take hours. Notice how "search a room for treasure" and things like that aren't even on the option list. Your reading of it is slightly out of context because it essentially says "if you take an action other than watching for threats[as your exploration action] then you don't use your passive perception to detect ambushes made on the party while you are travelling."
The section even says that you shouldn't use exploration turns when doing things like exploring dungeons room by room.
Context is important when reading rules.
The section on using stealth and perception just says that any time a creature wants to sneak up on someone they need to have cover and they need to make a stealth check opposed by the passive perception of anyone who can see or hear them.
If your attention is turned to something else, then it's not on noticing hidden threats. That makes sense to me. Why should there be a difference between keeping an eye out for hidden threats while travelling and any other time? If I'm rifling through the heap of broken furniture searching for treasure, I don't get to notice the giant spider sneaking up behind me. But the PC who opted to keep watch while I did that might. There wouldn't be an argument over this at our table because the players are aware that mechanics are only brought into play when there is uncertainty and the DM determines whether or not there is uncertainty.
Except there is definitely uncertainty in that situation. You may not be paying complete attention to everything going on. That doesn't mean it's impossible for you to detect hidden threats, just harder. As I said, you don't turn your ears off simply because you are searching. If the monster isn't 100% silent then you could hear them sneaking up on you. Some people have better senses than other people and are better able to detect noises or spot movement than other. That's the point of the Perception skill. I might give them disadvantage on their check if I felt they were distracted, however.
So, in theory, one person with high Perception skill might be rifling through the heap of broken furniture only to hear faint pattering of spider footfalls behind him and will turn around before it attacks. Another person might be so distracted by what he is doing that he never hears the spider. That makes there be some uncertainty as to the outcome. It depends on how stealthy the spider is and how perceptive the person in question is.
The DM can determine whether there is uncertainty, true. But in this situation it's fairly clear that there is. Which is why if I ruled that there wasn't, my players would feel that I unfairly ruled against them in an attempt to kill them. I already ruled this way once only to get half the players at the table very angry at me. It was a slightly different situation in that I ruled that an enemy didn't need cover to sneak up on them because they were all facing a different direction paying attention to a person they were talking with. I still had the person sneaking up on them make a stealth check. But my players were annoyed that a person was able to walk across the room in plain sight without being seen.
I can only imagine how angry they'd get if I told them they had absolutely no chance to hear the person either.
Edit: I will add that treating it this way creates a meaningful choice during play instead of just a matter how much a player chooses to pump Perception during character creation and advancement.
Technically, pumping Perception IS a meaningful decision. You had to choose it over another skill. Even then, that doesn't make you great at it, just good. In order to be great, you need to spent a feat or invest something like your Expertise choice into it. Those are options you didn't spend elsewhere. You should get some benefit from making these choices.
I feel like "Sorry, you didn't say you were looking for threats, you are automatically surprised" is a meaningful choice. It feels more like a gotcha.