If the perception check relies upon sight, then it is at disadvantage.
Relies is not a defined term in D&D. Common usage means that it must be dependent, not just utilized. As such, if you could not perceive the thing without seeing it, then the penalty applies. If you could notice it due to sounds it makes (or via other senses) the penalty does not apply.
So, a creature using darkvision in dim light:
* Tries to spot a creature that is hidden. The creature is stationary, and isn't making any obvious noises. I would rule that the only sense that could detect the creature is sight, so the perception check is reliant upon the sight, and thus the -5 penalty applies.
* Tries to spot a creature that is hiding. The creature has been slowly moving up to do a sneak attack on the observing creature. Because they are moving around, the observer has a chance to hear their movement and there is no penalty to the perception check.
* Tries to spot a defect in a tapesty. If they use their hands as well as their eyes, no penalty. If they use only their eyes - penalty.
* Tries to hear something. No penalty.
* Says they want to look around in a cave. I'd look at what is there to be found and determine what it would take for the PC to be using a sense other than sight to find it. Then I'd ask them to describe how they are looking around and would either apply the penalty or not based upon whether they are relying upon sight, or are using a different sense in a way that would assist.