Scouting is one of the main things PotC warlocks get to do, so let him do it, but there are ways to make success less automatic than you are describing. Others have already talked about Stealth vs. Perception contests. Here are some other ways to add uncertainty.
4. Oh no it's been detected. Warlock can just instantly recall it.
He can't instantly recall it. The spell states that "As an action, you can temporarily dismiss your familiar." Whenever you have opposing sides competing to take an action first, you should have an initiative roll. The warlock shouldn't automatically get to complete his action first.
Also, there are two interpretations of
Voice of the Chain Master. One interpretation, and I would subscribe to this one, is that the invocation expands the range of telepathic communication and perceiving with the familiar's senses from 100 feet (fourth paragraph of the Find Familiar spell) to the same plane of existence, but leaves the rest of that paragraph intact, including the part about being deaf and blind while perceiving through the familiar's senses. The other interpretation is that VotCM removes both the 100 foot limitation and the deaf and blind restriction. You didn't say how you were treating this but I wouldn't let the warlock fight, use skills, cast spells, etc while scouting with his quasit.
Hiding in plain sight: Another suggestion would be to populate your locations with monsters that are difficult to identify as dangerous if they aren't moving. Undead, constructs, and oozes are all good for this. Monsters that use camouflage, like cloakers, piercers, mimics and a lot of elementals will also be tricky. The quasit can see the fountain of water in the courtyard but won't be able to see the water weird in it. Put a mimic in the quasit's path. Either the quasit gets too close and the mimic attacks it, or the warlock tells the party he found an unguarded whatever and somebody else gets attacked by it later.
Blindsight: Even something as simple as giant spiders could pose a challenge to the quasit scout. They're stealthy, and have blindsight and a ranged web attack. A swarm of bats or giant wolf spiders would also work. These monsters could be fairly common in unpopulated areas.
Finally, I wouldn't let the time spent perceiving through the familiar's senses count towards a short rest for the warlock to regain spell slots or use hit dice. You didn't say that you were, but I thought I'd mention it.
As others have said, your player is in your game instead of other things they could be doing and wants to have fun with his character. He chose his pact in order to be an awesome scout for his party. More often than not, let him be awesome. But every once in awhile throw him a curve ball so that the outcome is not always certain. Yes, a random monster could walk up on the party but don't do that too often or it will seem forced. Look at his skill list and think of situations where the party would rather him be present and using a skill instead of scouting. Maybe he's the party face and they need him to nail a deception of persuasion check with an intelligent monster or NPC. Maybe there is a threat flying overhead and the party needs to hide under cover and make a group stealth check, but his roll is at disadvantage because he can't see or hear what he is doing. Maybe he is attuned to a magic item that the party would need him to use to avoid a dangerous situation. Even if these things happen rarely, your players will remember them and there will always be an added element of tension during the scouting trips and that should elevate the fun for everyone in the long run.
