It is an issue as you try to create consistency in your setting/story. You create a semi-believable illusion, the moment the illusion is constantly not believable or when you start assuming everything is an illusion, things go wrong.
Oh, me too. But then you solve the issue by making it true...