I think there does need to be some penalty for swimming in heavy armour, realism aside it just doesn't feel right genre-wise to have no penalty at all. A circumstance penalty equal to the base AC bonus* sounds about right, but set your swim DCs with this in mind. I don't think it's unreasonable to let the Paragon or Epic Fighter swim in scale armour with a suitable Athletics check. I recommend treating swimming in armour as a difficulty to overcome, not a flat ban.
Mind you, when Elric went overboard wearing the Dragon Armour of Melnibone, he sunk like a stone.
*Or go by weight carried.
Mind you, when Elric went overboard wearing the Dragon Armour of Melnibone, he sunk like a stone.

*Or go by weight carried.