Let's consider an example:
First, the DM will not ask the player of the character on watch to roll hundreds of Wisdom (Perception) checks for his watch, so elects to use the character's passive perception score of 15.
Next, the DM doesn't want to alert the party to the presence of the bandits, so is using the passive scores for the other three as well. Yet, those other three are busy doing other things, while also sometimes keeping a weary eye and ear out, while the first one is actively scanning and looking/listening for anything suspicious.
This is my issue...
I am perfectly happy with the DM using the typical results in lieu of asking for numerous rolls. But this terminology is not "passive" since the PC is actively using their skill repeatedly. As suggested, "standard", or "routine" or "typical" would be better suited to this.
I am also fine with the idea that the DM wants to use a roll of 10 when they want the result to be secret. Again, not "passive", but "secret or hidden" would be better terms IMO. The DM wouldn't want to "roll in secret" unless the PC was actually doing something relevant to the need for a roll at all.
Now, consider the three PCs at the camp. Doing other things actively while just being cautious of their surroundings. IMO these PCs are being "passive" in their perception. If they happen to pick up on something it is a bit of luck or chance. Now, that isn't to say they shouldn't have better chances if their Wisdom (Perception) is better!
I've long since played that the three PCs would have disadvantage on their scores due to the fact they are doing other things and not employing their perception in any sort of active or constant fashion. Their efforts are truly "passive".
In the above scenario the DM would want to keep the results secret (possibly?) and so could resort to only passive scores, but I would rule the three in camp have -5 to their scores. Does this make it less likely they will notice anything? Of course, they aren't on watch! However, there would still be a chance, it just shouldn't be as good as if they were actively on watch.