Cleon
Legend
I think I'd extend that restriction so that they lose all their special abilities while wearing armor.
I was thinking of allowing a bit more leniency, something like:
"A chameleon man can only dimension door if it has light encumbrance, is free to move, and is wearing no armour or light armour with no armour check penalty (provided the chameleon man is proficient in the armour)."