If it is just a role playing issue for you, then instead of them actually wearing leather, maybe say the feat is in reality a special ritual they know that casts a permanent armor spell on their regular clothing that grants the same bonus. then they get the benefits, but can still keep the look...