So, do you have:
Passive Acrobatics?
Passive Animal Handling?
Passive Arcana?
Passive Athletics?
and so on?
If not, why does Perception deserve a "floor" and not the others?
Any of those? No. Not because I couldn't use them if I wanted to... but because those are all skills wherein the PCs do not have to have a contested check where the opposing person (or thing) is being done secretly.
However... the four I do use Passives on are Perception, Investigation, Stealth, and Insight.
For Perception it's easy... if there are people / living creatures hiding out there somewhere, they have all made DEX (Stealth) checks and thus are hidden to a certain extent. When the PCs arrive, if any of them have a Passive Perception higher than a creature's check, then the PC notices them without needing to roll. So in any group situation, each PC might see some (or none, or all) of the creatures hiding. If the player suspects there might be more creatures out there for their PC to notice, they might then ask to make a Perception check. They then roll the check as normal. As they have already been told which creatures they saw based on their PP (which is essentially a 10 on a die roll)... if they roll a 9 or less they know they haven't seen anything more than what their PP told them, and if they roll 11 or higher (plus then their mod), they might find any other creatures whose DEX (Stealth) checks were higher.
For Investigation... it's important to bear in mind of my own personal house rule, which is that Investigation is used to find any non-living thing that is hidden and needs to be discovered-- secret doors, traps, items in secret compartments, etc. In this case it's the same process-- any object that has been hidden has a DC to be noticed. When the PCs arrive, if any of those things have a DC lower than the PC's Passive Investigation, then they notice it right away. They see the tripwire across the hallway, they see the edges of that poorly-concealed secret door, they see the bulge in the rug where someone hid the doodad. But if they arrive and don't notice anything, but suspect there might be something to find... then they can state they are searching the area. At that point it comes down to either them describing what/where/how they are looking (wherein they could find the hidden thing without even needing to roll), or they could say they are making a general search of the area and thus make an INT (Investigation) check. And again... any die roll lower than 10 produced no new info over what their Passive Investigation gave, and anything higher might discover a more well-hidden thing.
For Stealth, it just saves me time and energy when the group is on long journeys and request to move at a slower pace in order to reduce their noise levels as they march. So rather than everyone rolling DEX (Stealth) checks repeatedly over however long the journey is, I just use their Passive Stealth as the DC for any other creatures they might march past who have a chance to hear/notice them using
their own Passive Perception (if those creatures even care.)
Finally, for Insight... this tends to be a rarer thing but I still use it on occasion. If an NPC is trying to hide their intent but I know full well they aren't good at it (either by fiat or because I rolled a CHA (Deception) check for them and rolled poorly)... I will just tell any players whose PC is interacting with the NPC and whose Passive Insight is higher than that check that the NPC is obviously being deceitful in some manner. More often than not though it ends up being unnecessary because the players tend to be a suspicious lot and will ask to be insightful on their own.
Other than these four skills... I have not had a situation where having a Passive number for any other was necessary.