The most common mistake originates from the fact that the most common way to see if a PC is surprised is to roll Perception. Therefore, if you are not surprised it is because you have perceived something: body language, verbal/somatic components, heard them breathing, seen their shadow, whatever explains the established fact that you have perceived them via a successful Perception check.
However, immunity to surprise
does not equal a successful Perception check! Just because you are not surprised, this does not mean that you have perceived...anything!
It just means that you don't suffer the game mechanics of surprise: cannot move or act on your first turn, cannot use reactions until after your first turn.
The situation: an evil caster wants to cast a spell from hiding.
First: establish surprise. Everyone rolls Perception (because the DM feels that this is the appropriate method to establish who is surprised in this situation).
The PCs who's Perception rolls exceed the villain's Stealth perceive the caster; make up the exact reason why on the spot. Maybe they heard the verbal component, whatever.
Those PCs who rolled too low on Perception did not perceive, therefore they are surprised.
How does immunity to surprise come into it?
If the Alert PC succeeded on their Perception check then they heard the verbal components and realise there is an enemy caster round the corner. His roll means that he wouldn't have been surprised even without the feat, but his Perception check tells him: caster, around that corner.
If the Alert PC rolled too low, then they did
not perceive the enemy,
even though they are not surprised! The alert PC did not hear the verbal components, therefore does not know there is a caster or that there is an enemy around the corner! However, he may still be first in the initiative order and can still move/act on his first turn and may take reactions even before his first turn.
So, Perception checks are rolled and the Alert PC fails. Initiative is rolled and the Alert PC goes first.
DM: So, Alert guy, you're up. What do you do?
Alert PC: I go around the corner and attack the caster.
DM: No, you do not know there is anyone there.
Alert PC: Okay, I have a bad feeling about this, so I move behind the paladin and take the Dodge action.
DM: Cool. It turns out that the Alert feat hasn't destroyed my game after all.