I wouldn't spend my time coming up with a subsystem for swimming in armor. The task resolution mechanic (ability check) handles this fine, if you happen to think the act of swimming in armor has an uncertain outcome. Perhaps in a charged situation where time is of the essence and danger other than drowning abounds, a check makes a lot of sense given the goal and approach of the character. In other situations, where there is no other immediate danger or drama afoot, then no check is required for just swimming in armor.
I think that exhaustion on a failed check is a suitable cost, absent additional context about the scene that involves swimming in armor. For example, the character swims ashore or back to the boat or the person he or she is trying to rescue, but takes a level of exhaustion in the process.
It needn't be more complicated or granular than that in my view.