I know you weren't talking to me, but I think I get what Jeremy meant by
floor.
The only passive I use, besides passive Perception, is passive Survival. I like to do this for navigation checks because I want to keep the check result hidden. I roll for the terrain as a contest against the navigator's score. A success means the party went in the desired direction whereas a failure means they went in some other direction which I determine randomly. I don't want the players to know which is which, so I keep the result secret. The upshot of this is that the navigator has a "floor" which might let them reliably navigate through certain terrain types and conditions.
I don't think it's a matter of deserving. I can only speculate that Jeremy was explaining the rules with an orientation towards combat, but completely failed to make that clear. Here's a
link to the infamous podcast where he says "floor" at 23:39. However, at about 11:39, near the beginning of the Sage Advice segment, he states that a "mechanistic" approach to stealth, which he typifies as that presented in 4th Ed., works "primarily" in combat. He says this while prefacing his remarks on the mechanics of stealth in 5E by talking about why the decision was made to put stealth more firmly in the hands of the DM.
So basically, Perception has a "floor" in combat because everyone is alert to danger in combat unless they are incapacitated or the like. Outside of combat, having a "floor" depends on whether your character is alert or whether some other task is occupying their attention.