Ironically, I think if you're up-front about using illusionism with your gaming group, that's fine; then they can either tell you that's not what they want to happen or take a walk or whatever upfront, too.
But a lot of GMs don't want to, well, break the illusion, and want to act like they're...