Yes, it is correct. To rely on something, it needs to be the primary sense used.Which is not correct.
Examples of the absurdity created by your interpretation:
B is trying to find A (via the Search action). In each of the following examples, A is a Wood Elf, and is currently hidden [unseen and unheard] and wears a Cloak of Elvenkind.
1) A is hidden, and peering around a tree, 30' away from and observing B, preparing an ambush.
2) A is hidden, 10' away from B and invisible.
3) A is also a Warlock, and is hidden in the radius of a Darkness spell, 30' away from B.
4) A is hidden in rain (light obscurement) and 30' away from B.
Using your logic, B has disadvantage on perception checks to locate A, when A is unseen but visible (hiding behind a tree and peering around, or hiding in a downpour), but OTOH when B is totally impossible to be seen (invisible, in darkness) there is no disadvantage, and the Perception check is made normally (as per the Search action and locating hidden creatures).
In other words, your interpretation of the word 'rely' leads to absurd outcomes Mine doesn't. My interpretation is therefore to be preferred.