This is false. Armor is rarely all that heavy if you aren't exerting yourself much, and are in good shape. Only badly made armor feels heavy enough to be annoying. It certainly isn't cumbersome, chafing, or annoying to wear, unless it literally doesn't fit you or is poorly made.
It is most certainly cumbersome. It's not nearly as cumbersome as a lot of people used to think (before the SCA and modern Armored Combat Sports and historian YouTubers and such debunked misconceptions about it being enormously heavy and unwieldy), but no one wears armor all day long unless they've got a good reason (and yes, sometimes that good reason is a hobby; but hobbyists do it on special occasions, not all day every day).
Now, the biggest factor in this whole argument is probably just the DM and players being on the same page, and not inadvertently getting into an unfun adversarial dynamic.
Players should understand that in civilized areas that the same town guards who enforce weapons and armor limitations on them are enforcing them on random thugs and other common threats in towns and cities. Encounters in those areas will be
different, but PCs shouldn't feel inherently screwed by not being equipped the same as they are in a dungeon.
The tricky bit here is how modern D&D has enabled non-armored PCs with big Dex bonuses and so forth, which minimizes the disparity in AC between heavy armored and light armored characters to mechanically better support light-armored, swashbuckling character concepts, but this has the unintended secondary effect of ironically making them BETTER in a fight in a social circumstance where armor is inappropriate than the front-line fighters whose characters are built to use heavy armor.
As folks have complained in this thread, it feels sometimes like shield-wielders get screwed by the action economy, since they need to spend an Action to ready their shield.
This stuff also happens with encounters in the middle of the night when PCs are resting, of course. Heavy-armored fighters and shield wielders suffer a greater disparity between their regular combat capabilities and their "surprised in the middle of the night" abilities.
As a DM, you have to think about what you're trying to achieve and work with and communicate with your players. Bear in mind that a lot of fights SHOULD have some forewarning, and that naturally readying your shield is going to be the first thing shield users do in their round of warning. And let them know that in town their opponents will usually be similarly impaired in terms of arms and armor, so they don't feel so put upon.
Also consider mechanically incentivizing Strength-based combat characters in some way, if you're finding that all your players are opting for Dex-based characters and you don't like what the game looks like when they do. Whether that be by just using the Variant Encumbrance rules (I do), or by working in a few house rules. E.g., maybe give heavy armor free Damage Reduction like the Heavy Armor Mastery feat, so the plate-armored guys feel like extra badasses when kitted up, compensating for them being more vulnerable than the Dex fighters when dressed down in town.
but also if you have high CHA... 
