Hi. I'm an armorer of sorts. That is, I teach classes in how to make chain armor, usually at Renaissance Faires.
Padded armor wasn't meant to be "armor" in any real sense. It represented the padding normally worn under the more substantial armors.
Chain's main purpose is to turn a cutting edge into a blunt collision. If the chain is all you're wearing then instead of being hit with a sword or axe it's more like getting hit with a crobar or sledgehammer. And nobody likes getting hit with a crowbar or sledgehammer.
So the padding is meant to soften the blow. Now considering that the heavier armors were more common in England (a cold country) and northern Europe, wearing an extra layer of sheepskin or a vest that looks like a double layer mover's pad wasn't going to give you problems from the heat.
Chain has the odd property: As it pulls dow longer, it draws out narrower. Over the course of a day you can feel it pressing in on your ribs, and it actually can impair your breathing. Sleeves don't fit your arms well. They can't taper towards the wrist because you hand has to be able to fit all the way down. Further, the other side of that "longer means narrower" thing is that, if you're holding your arm out at all then gravity will pull it wider, and thus shorter. I can easily see how baggy steel sleeves could be a problem when casting spells.
Scale and Banded armors are similar to chain in many ways, but the "longer=narrower" thing isn't there, and it's generally less flexible.
Now the only plate you'll ever want to wear is a set that's fitted to you, and fitted well. Properly done, the weight ends up resting on your hips more than anywhere else, so yes it's not bad to ride in. But keeping your balance when on foot, and someone's swinging at you? That takes skill and practice (i.e. armor proficiency.)
If you do fall down, do you know how to get up, quickly? It isn't easy, and yes there's a skill to it. Again, armor proficiency.
Now consider dancing in any of these. That's the best way to characterize fighting in the stuff, as you advance, dodge sideways or backwards, duck and recover. And on most battlefields that's on uneven ground. and the better the armor, the more top-heavy you are. If the armor includes leggings (the difference between "Chain Shirt" and "Chain Armor") then your legs are heavier and a bit harder to lift. Easier to trip or stumble when engaging in that dance.
I've seen the videos of people in chain, scale, plate etc doing shoulder rolls, running, mounting a horse and so on, and they make it look easy. I know from experience that it isn't. Those people had a lot of practice and more than a little training. Their armor had to be properly fitted, or even custom made, for them to do that. (With plate or breastplate the armor may have had two or three trips back to the smith for adjustment and refitting.)
If you've ever ridden a horse you'll recall how high your foot has to go to reach the stirrup. It's almost chest high on most people. Now think about doing that while wrapped in protective steel worn over a full snow suit. (the padding). Next, from that contorted position that has poor leverage, hoist your weight up with that one leg. And remember how much the armor weighs. You have to be strong to heft yourself and plate armor or even just breastplate up from that position.
D&D says that a person can put on "light armor" like a chain shirt in a minute. If all you're doing is pulling the shirt on, think more about six seconds. It's a straight pull over. But putting it on for battle means putting that padding on and cinching it to a good fit, then adding the armor, then adding a belt to take some of the weight off your shoulders, so it doesn't pull down/in and make it harder to breathe. Knowing how to arrange the whole rig is also part of "proficiency".
Finally, remember that the armor is presumed to include gloves or gauntlets of some sort. Leave those off before a battle and your friends end up calling you "stumpy". Now try to move your arms like a ballet dancer, to emulate the arm movements needed for the somatic part of spell casting. Hold and manipulate the material components just right, while wearing those gloves.
Alternatively, unstrap the gloves (they have straps and buckles that secure them to your wrists), while wearing gloves, before each spell, then secure them in place again. You'll get spells off every two, maybe three rounds, at best.
