By the words of your very argument, this is in error.The -5 to Perception in dim light only applies to Perception checks to see stuff. It doesn't apply to Perception checks (or passive perception) generally (which rely on all 5 senses)....
A given area might be lightly or heavily obscured. In a lightly obscured area, such as dim light, patchy fog, or moderate foliage, creatures have disadvantage on Wisdom (Perception) checks that rely on sight.
If a general perception check relies on all 5 senses, it relies upon sight (in addition to the other senses). It is only when sight does not play a factor, at all, that it is not implicated.
A PC can elect to focus on only one sense when making a perception check, but when passive is relied upon, they make no decisions. If the perception opportunity would benefit from sight (as in there is a potential to see some clue), and it is in Dim Light, then it is hampered with the -5 penalty (subject to the always present DM override).