The thing is that the familiar is a NPC. So the DM can do whatever he wants there, especially since for example the stealth check (if any, the DM can take into account passive stealth if he wants), vs. a passive perception, etc. Even if the DM is playing things fairly, there does not need to be anything visible from the player's side.
And as you point out, if fairness was the point, the DM also had at its disposal all means to explain why it was fair, maybe after the game. Instead of that, there was a massive perception of unfairness, and an overall sentiment from the players that it was the case of "I have a a kobold ambush prepared, there is no way a stupid familiar is going to spoil this".