If I'm understanding you correctly... you're suggesting a situation where someone is invisible but no one is there to sense him. Sure, in that case I wouldn't ask for a DEX (Stealth) check, just like I wouldn't ask for someone standing alone in a room to make a DEX (Stealth) check, invisible or not. If no one is there to sense him, then he is by definition 100% beyond being sensed.
But if someone is invisible and the possibility exists that someone could walk by and sense him (through Passive Perception)... then I absolutely would ask for a DEX (Stealth) check right at the top so that I'm prepped for when the passerby goes by. Now depending on the situation and where the invisible person was standing I might give Advantage on the DEX check or perhaps even some other additional bonuses... but they still need the check just to see how well or poorly they're holding themselves. If you're invisible but you still stand in the middle of a room with people walking by you, that makes you more prone to being noticed. Or if you try and regulate your breathing but for whatever reason are breathing heavy, that also will make you prone to being noticed. That DEX check tells me how well you're doing.
For me, just being invisible doesn't give you a free pass to never being sensed, even if someone isn't specifically trying to find you. The one thing the Invisible condition gives you is satisfying the Heavily Obscured requirement regardless of where you happen to be.
I get what you are saying, I just think that a trained Stealth invisible PC should only be rolling if there is a reasonably decent chance of being found. The invisible PC is in a room looking for stuff. A maid opens the door. The PC freezes. No need for a check in my mind unless the maid walks directly to him (through his location, whatever). I do understand that in this scenario, the DM might want to add to the suspense of the scene, especially if it is a small room and he asks for a check. But a large banquet room? I probably would not even bother.
And even if the maid were to hear something, without seeing something, she would probably just shrug it off and go on with her day.
The vast majority of NPCs shouldn't be suspicious of an occasional strange sound or odor or whatever. In the tavern's beer cellar and the stableboy hears a sound? It's probably rats or the creaking of the floor boards above. I wouldn't expect most NPCs to suspect invisible creatures.