That's all the more reason to confuse the issue. Glamer your armor to look like robes. If possible, get some spellcaster-friendly armor (Twilight mithral shirt, for instance) and Glamer it to look like heavier armor. Have the spellcaster wear a monk's robes and carry a sai or nunchucks. Have the monk wear mage's robes and stick a gem on the end of their quarterstaff. If you're a Psion, wave your hands and incant random sayings whenever it's convenient, so people think they're necessary for your powers. Alternately, hide your power displays and pretend to be a hireling cowering in the back.
Plus, illusions.