On a side note, this is one of those verisimilitude issues that usually kicks fighters right in the sweet spot while leading mages, warlocks and other caster-types alone. It just leads us down that endless path of burdening the martial types with reality while further empowering folks who can already wrap reality around their pinky.
Maybe.
What I find is that in practice, verisimilitude regarding 'epic adventuring' tends to kick heavy armor brutes to the curb, but not necessarily fighters and other mundane heroes. It's not merely swimming that presents a challenge to a would be heavy armor wearer, but anything outside of battle on a level plain or combat on the tournament field. He's slower. He's more likely to trip and stumble. He tires more easily. He has greater difficulty climbing, swimming, and squeezing through tight paces. You are more exposed to deadly heat and humidity and metal armor can be very uncomfortable in the cold. The jungles, caves, ruins, chasms, and mountainous slopes were adventuring takes place tend not to be the sort of places where you want to wear very heavy armor. As a result, most players in my game of any class tend to prefer to avoid heavy armor and compromise on some lighter form of protection. Giving up a few points of AC in exchange for greater mobility usually works out as a good trade. It's not necessarily realistic that this lighter armor is less problematic, but it does have verisimilitude and we do know that real world adventurers - say Cortez, tended to prefer lighter armor.
But favoring light or medium armor over heavier armor doesn't necessarily favor the spell-casters. Clerics end up suffering more than fighters, because unlike fighters, strength is usually a 4th stat for a cleric and dexterity a 6th stat. As such, they can't make up for the problems associated with armor nearly as much as a fighter can. Indeed, since fighters can more readily afford to put points in skills toward swim, climb, and jump and really any other physical skill than a cleric, it's usually fairly trivial for a mid-level fighter to counteract whatever penalties come with his armor. (Under my house rules, this is even more true, so that it's certainly true that one way that mundane classes like fighter, rogue, explorer and hunter excel spellcasters is that they have more reliable mobility with no reliance on spending spell slots to make up for physical shortcomings.) And indeed, since classes like Wizard and Sorcerer likewise can't afford to spend on physical abilities and wear no armor, saying that they aren't as penalized when swimming is making a virtue out of a problem. It's not unusual for the sorcerer with no ranks in swim and an 8 strength score, to have more problems swimming than the fighter in his much heavier armor - after all, the fighter with a +13 or higher swim score could swim like an Olympic athlete (if not Aquaman at higher levels) if he took off the armor. And a sorcerer that spends spells to swim, improve AC, and otherwise defend his person from harm is generally leaving himself with reduced combat options and at risk of running out of spell slots.