Sure, but from a game rule mechanics point of view how does someone know. Like you said, someone could pretend to cast a spell and then try to touch them. Besides, with 3E, you don't necessarily have to look like a typical spell caster to be a spellcaster. What if they cast a still, silenced...