You sleep "fine"? Are you using a sleep tracking device to see how good the rest you're getting actually is? A lot of folks function chronically under-rested, and it's definitely not good for us. I don't mean to get into personal stuff or give you a health lecture, but it's very common for folks to underestimate how bad their sleep is.
I have a sleep number bed, it does indeed tell me how well I sleep. At first it was 10-15 points below my average, once I got used to the situation I've been sleeping fine.
I agree that how realistic we want to get with this stuff is a matter of subjective preference.
If I want a quasi-realistic game I'd probably say sleeping in gambesons is ok, but nothing heavier. And folks would have to deal with somewhat compromised protection (grab your helmet and shield and go!) during night attacks.
From a player perspective it results in vast disparities. I will never understand the penalties DMs put on PCs in heavy armor because they're 5-10% less likely to be hit by a weapon attack. I've run into DMs in the past that simply hated heavy armor and would go out of their way to screw over people that used it.
In any case, it's just a pet peeve. If I find myself in a game where I'm penalized for wearing heavy armor I just won't play a strength based PC unless they're a barbarian.