Also from DnDBeyond:
Passive Checks
A passive check is a special kind of ability check that doesn't involve any die rolls. Such a check can represent the average result for a task done repeatedly, such as searching for secret doors over and over again, or can be used when the DM wants to secretly determine whether the characters succeed at something without rolling dice, such as noticing a hidden monster.
Here's how to determine a character's total for a passive check:
10 + all modifiers that normally apply to the check
If the character has advantage on the check, add 5. For disadvantage, subtract 5. The game refers to a passive check total as a score.
So, yes, you do have disadvantage to passive checks. It's just calculated as a -5 to the passive skill. My mistake was applying it to all perception, not just sight.
Edit: the underlines are me, not dndbeyond.