Well, if you are sleeping in armor, there is a good chance you may not be taking it off often enough, and the effects of continuously wearing it would start to surface. So you could create penalties for not finding time to remove it, replace underclothes or let them dry, doing hygiene, etc...