Beleriphon
Totally Awesome Pirate Brain
The difficulty of detecting a hidden creature is based on its Stealth check, which it hasn't rolled if it didn't take an action to hide. Modifiers may apply due to circumstances, sure, but the Stealth skill of a target is always the primary factor. That's the mechanical irregularity which would be introduced here.
I have a question. How many creatures that turn invisible aren't trying to hide somehow? By default maybe the invisible condition should just trigger a stealth check by default.
Some default invisible creatures (invisible stalkers for example) still need to make stealth checks to sneak up on a character. Just like they would have to make a stealth check to escape combat, the difference is to escape combat they don't need to become unseen by breaking line of sight.