I think you set up your stats the right way.
Personally I'd dump Cha totally, leaving it at 8. You can use those two points better elsewhere.
Main stats should be Int and Dex for a melee oriented caster (or Dex and Int for a casting oriented fighter, respectively).
Maybe you could also drop Str by 2 points if you chose to go with weapon finesse. Yes, that'd weaken your damage noteticebly, but a gish surely has other means than strenght to deal damage (spell storing weaponfor instance, if your dm wont allow arcane strike).
I personally dont like DD very much. Dunno why, but I prefer a build as "plain" as possible without having to be a "freak"
But yes, it obviously offers some possibilities to get some str and bonus spellslots - even though your spell levels wont be progressing with it.
Oh, btw, perhaps you want to take a look at the duelist PrC. Its has quite the same style as Swashbuckler and provides some interesting benefits. Int to AC. When fighting defensively, another +1 to AC per Duelist-level. Full BAB.
Still, no spell-progression.
So I think you have to decide what you want. Do you want to be primarily a fighter who augments his fighting with a bit of magic to do more "ouch" and "argh!"? Or do you want to be a caster primarily, who wants to dabble into melee combat once in a while?
Imho the only way to achieve a good balance between melee fighter and caster is by taking and focusing on eldritch knight.
But be warned. Playing a gish can be very annoying when youre at low levels. But when you come to mid-level, they begin to rock