And that's enough for you to say that no matter what you roll, there is no way you could possibly hit the invisible creature?
I don't find that realistic, personally. You should get a penalty, sure, and debates can be had about how big the penalty should be--but hitting the target should be...