Let me see, I've slept with multiple socks, sweatpants over jeans, a couple of t-shirts, shirt, jacket, hat ... basically every item of clothing I owned. Oh, and it was in a dormant volcano so sleeping on volcanic rock.
I'm not saying it was the best night's sleep I've ever had but after most of a day hiking to get to the top of the volcano I was fine if a bit stiff. I think people who have never had to sleep in adverse conditions overestimate the need for a soft comfy bed (that, again, most people throughout history have not had access to). As long as you aren't miserably cold and wet or incredibly hot I've always been able to get a decent nights sleep when I needed it.
It's just one of those assumptions, like people in full plate needing a crane to get on a horse. I know for a fact that you didn't need a crane, I sincerely doubt you couldn't sleep in armor if you've been active all day. Of course if you were in what you consider a safe place you're going to get comfy. Other than that? It's always been a weird thing to penalize PCs for considering everything else we ignore.