There's a 2 foot tall invisible lawn gnome statue on an otherwise clean shelf.
- No dust, no cobwebs, the shelf is not bending under the weight of the statue, nothing.
- You don't have a bat's sonar.
- You don't have keen enough hearing to hear the air molecules bouncing off of it.
- It doesn't have a unique smell, it's not making any noise.
- It's not sparking magic gnome dust to give away it's position.
- Its' a clean room and there's not enough dust floating in the air to give away it's position
- There's no pigeon sitting on top of it, no spider crawling over it or anything else
How could you possibly notice it unless you happen to bump into it?
But my answer included examples that ran the gamut, from being undetectable without special abilities to it being obvious there was something you couldn't see with options in between. The undetectable statue is just one possibility of many.
I don't understand why it bothers you that there's a chance the DM may show some creativity and think about how people would perceive the world around them.
The "invisible object in the room" has featured in more than one game I've played, and is a pretty common trope of fantasy/sci-fi/comics.
It's also off topic of creatures running around in combat turning invisible, and is something simply not covered by the rules. When the rules are silent, I use my best judgement. I can't think of anything else to add.