I think it would be interesting to have an encounter or two with the PCs being unarmored. It would change the balance a bit in the party and might lead to some interesting tactics. The enemy just needs to be designed with that in mind. I wouldn't throw the same sort of encounter at a party under those conditions I would when they are all fully prepared.
Yeah, it's not fun. My group is currently playing in KotS and the rest of them decided we needed to take a rest inside the dungeon. The DM said we had to remove our armor, which I didn't really like...but that's what he wanted.
An hour into our rest a patrol comes along and we have to prepare for combat. Thanks to the Eladrin in our party, he gave us a one round warning, which allowed me to at least put on my shield.
I play a Fighter and normally, I have a 21 AC. It's 10+1 for half level, +7 for Scale, +1 Enhancement and +2 Shield. Without armor, I had a 14 AC (I had a 12 DEX which at least gave me a small bump). This equates to almost an auto-hit. Our Cleric burned Priest's Shield, which bumped me to a 16, and mean that now they hit on like a 6 or 7.
Initially, I did my best to try and hold the door into the room, but after a round I was already bloodied and close to dead. I had to shift+move so that I could move to the back of the room and
hide behind the Wizard. There was nothing "fun" or "interesting" about the encounter...in fact, we were pretty close to a TPK. The only thing that saved us was the fact that pretty much everyone burned through their dailies to help us survive. After that, we donned armor, exited the Keep and camped outside.
Honestly, if that's the way it's going to be from now on, I might insist on always sleeping in town since I was constantly worried that another "patrol" might come outside the Keep.
The additional problem is that right now we're only at level 2, so only about half the party even has magic armor. At higher levels though, when everyone is wearing stuff like +3 or higher, a
significant portion of your AC is now coming from your enhancement bonus...something you don't get when not wearing your armor. So a level 30 PC in +6 Godplate? He'll be -20 AC for the encounter when out of his armor. That IS pretty much a scenario where the enemies will hit on anything other than a 1.