I really think people are glossing over how powerful the non-hostile charming aspect is. You win an opposed check, to gain advantage on all social rolls for a minute. Which means once you charm the target, you gain advantage to keep charming them. There's no downside, and no expenditure of resources. Charm Person requires a spell slot, and the target knows it was charmed when it wears off. If they didnt want casters dominating social challenges with magic (which is a good call btw), I think it is reasonable for swashbucklers to not do the same. And watch out if they took persuade or bluff as an expertise skill.
I'm not sure the best way to tone this down. Letting it not be reapplied seems fairly appropriate, as it lets the swashbuckler make a good first impression, as opposed to being Grima Wormtongue cranked up to 11.