Charisma represents more than just social skills: it represents confidence, self-assurance, and a certain kind of inner strength. Bards, paladins, and sorcerers all draw on it. It's not too much of a stretch to say that for certain kinds of fighting traditions it can make a real difference. Maybe it represents a certain difficult-to-define virtuosic edge that you get when you completely believe in what you're doing, and give yourself completely to the flow of the fight.
If you don't buy it, though, it's just fine to limit the chosen ability to just Intelligence. There might even be very good mechanical reasons to rule out Charisma, since paladin/duelists could be very, very powerful. One of the nice things about the Int-based duelist is that apart from the duelist and the Int 13 prerequisite for certain feats, Intelligence is unique among the mental ability scores in having very few combat applications under the core rules.