Like I said, it's not a simulation. If people want it to be a simulation they can make more detailed rules. Just like when you are paralyzed, your AC still includes your dex bonus. A more simulationist approach would include rules for not including dex in some circumstances and not including armor in others (like touch attacks). But it's a game. Generally simpler is better.
And yes...the simplest thing is just ignore armor and there is nothing wrong with that, either.
In the case of my group, the PCs and their NPC cohorts are travelling in boats. No one is carrying anything other than armor and weapons and the like. No one is carrying around their packs. They are stowed. Of the two characters that fell out of the boat, one has no armor and a +5 athletics. The other is wearing scale mail and a +2 to athletics. It would be off to have the one weaker armored character not have some greater difficulty. So if I call for a DC 10 swim check to stay afloat through some rapids, it seems fair that the guy wearing scale mail will have a DC 14 check or make the DC 10 check at disadvantage (I'll probably just do the latter...again, simpler is better).
My assumption is that when combat occurs or other dangerous situations, people are dropping their encumbering gear or have left it behind in a camp or at some other rally point. If your group is doing detailed encumbrance by the book, then yeah...it makes sense to use weight. I'd say if you are heavily encumbered, you have disadvantage on swim checks and you can drop gear to lighten up. If you are unencumbered, swim checks are at advantage. Seems fair.