The first swashbuckler in my campaign has just died; foolishly scouting ahead on his own, on the 3rd level of the dungeon (he was level 2), when he was on 4 hit points and the party was all wounded and out of spells. He encountered a ghoul, it hit him for 5 damage, and that was that.
I think the lack of heavy armour is fine, even without extra bonuses. In fact, considering that a Swashbuckler really benefits from a high Dexterity, heavy armour would be more a hindrance than a help.
What might hurt is the loss of a shield or buckler, but some of the Two-Weapon Defense feats would mitigate that. Assuming that the Swashbuckler's main ability score is Dexterity, there's likely to be a +7 or +8 bonus from that at higher levels, just as good as what the heavy armour would give him. Adding a +5 chain shirt just makes things better.
The design idea behind the swashbuckler - it's primarily a fighter, so no good Reflex saves, etc. - is a very interesting one.
Cheers!