My group had an encounter where we got ambushed in our sleep...it was one of the most difficult encounters we ever played because I (the Fighter) had an AC of like, 13 (10 + 1 for 1/2 level, +2 after I put my heavy shield on) IIRC. I simply COULD NOT do my job as a Defender with an AC that low. I ended up running away from the main fight with a few hit points left to hide behind the Wizard, because he had like an 18 or 19 AC. It completely ruins any sort of encounter balance and totally screws players with Heavy Armor. I'm seriously considering getting Summoned Armor just to avoid such an issue in the future.
Plus, there's also the fact that we were level 2. If you were to ambush a level 30 character in no armor, it would be different...and by different, I mean worse. A Fighter wearing +6 Godplate gets 20 points of AC from his armor, all of which is now gone. That's a HUGE punishment compared to the Light Armor classes (most of whom would've been within a few points of the Fighter's AC anyway) get no penalty whatsoever. While it might be "realistic" (and even that is debatable as there are stories of people sleeping in armor), it's simply unfair and tosses balance out the window.