The Crimson Binome
Hero
In my experience, the most common cause of such "meta-gaming" is simply a lack of communication between the player and the DM. Even if the character shouldn't know about the possibility of invisible monsters, the player doesn't know that the character shouldn't know that, because the DM failed to adequately describe the setting.Okay, think of it in terms of the movie Inception. The idea of an invisible creature is planted inside the player's head because she hears it at the table. Her character would not have that clue within the game world. The belief comes from the inception placed into the character's head from an outside the game source. That's why I'd say a die roll would be a better determiner.
Or you could compare the situation to a player who knows germs cause disease letting their character believe the same.
The question of how much the characters know about the setting beforehand is a significant one, which can greatly change the tone of a campaign. If you're playing in the Forgotten Realms, then it makes sense that everyone would immediately assume invisibility, because spellcasters are all over the place in that setting. If you're playing in a setting that's more historical, then perhaps nobody would think of it unless they'd encountered it before. The easiest way to resolve this question is usually to make a check, using the appropriate knowledge-type skill, where the DM can set the DCs as appropriate for the setting; that way, it only takes a brief conversation at the start of the game to get everyone on the same page about what can be assumed and what requires a check (and how high those checks are likely to be).