If you're not trying to see them, the
n of course perception checks to see them automatically fail.
Think it through for a second.
Bob, Rogue 3 (Stealth +7) has a Cloak of Elvenkind, and is engaged in battle with
Terry, the perceptive Fighter 3 (Perception +5, Passive 15)
Example 1:
On Bobs turn, he drinks a potion of
invisibility as an Action, and then takes the Hide action (he can do this at any time, thanks to being
invisible) as a Bonus Action (rolling his Stealth with Advantage thanks to the Cloak). He scores a result of 20 successfully hiding from Terry, then slinks away 30' with his movement.
His opponent Terry then takes the Search action on his turn to locate Bob.
His Perception check is NOT at disadvantage because he cant SEE Bob. He rolls a total result of 21 and notices the long grass they are fighting in being pushed aside by Bob as he moves, giving away Bobs position.
He Action surges (gaining another Action) and shoots his longbow at Bob (at disadvantage due to
invisibility) once and misses.
Example 2:
On Bobs turn, he fires his bow at Terry (and misses) with his Action, and then ducks down into the long grass they are fighting in, and takes the Hide action as a Bonus Action (rolling his Stealth with Advantage thanks to the Cloak). He scores a result of 20 successfully hiding from Terry, then slinks away under the cover of the grass 30' with his movement.
His opponent Terry then takes the Search action on his turn to locate Bob.
His Perception check is NOT at disadvantage because he still cant SEE Bob,
and even if it WERE possible to see Bob in the grass, Terry is relying on other senses in addition to sight.
He rolls a total result of 19 and fails to locate Bob in the grass. He backs off with his movement, and uses his Bonus action to Second wind and recover hit points.
-----
In a nutshell the Stealth bonus of the cloak is available at all times.
The Perception penalty it confers is much more circumstantial, as generally when you're Hiding (using Stealth) you cant be seen anyway, and even when you are, the Searcher is relying on other senses to locate you in addition to sight.