Perhaps use the exhaustion mechanic? Off the top of my head, I'd give a level of exhaustion on a failed save or a specific amount of time (one minute?)
I love this idea. You can be as punitive as you like; con saves every round, every minute; whatever.
In fact I'm totally using this if it ever comes up. I'll just apply disadvantage on the Athletics checks themselves as per stealth, but swimming in any armor will require con saves or incur levels of exhaustion.
I could get granular with it, too. Save 1/round in dangerous, choppy waters or 1/minute in calm water. Save is made with advantage if in light armor, disadvantage in heavy.
Maybe a con check instead of a save, so the bonus stays low, but make the DC on the easy side, like 10. If con save, probably DC 15.
In actual play I think this results in the average low level fighter in plate mail who falls into choppy seas floundering around for a round or two and then drowning. (Disadvantage on athletics to go anywhere, disadvantage on a con check DC 10 each round to avoid exhaustion, and after a few levels of exhaustion his speed hits 0 and he drowns basically for sure.)
I'm okay with this. It allows for practical action, with steep consequences and adversity.
A medium armor master champion fighter in half-plate, on the other hand, will probably be able to swim to safety, provided it isn't too far away.