This guy tried it:[video=vimeo;13634653]https://vimeo.com/13634653[/video]
Well that answers it

When I did lifeguard training we had to swim wearing shirt, jeans, sneakers and this was difficult for all the trainees who were in peak swimming condition.....wearing even leather armor....yeah right!
As was suggested up-thread the cheap and easy response is make tests at disadvantage and give armors over light a higher target number as well. The weight of that guys armor was such that it cancelled his natural buoyancy allowing him to walk on the bottom of the pool.....he as they say "sank like a rock"