So my question is level progression. Should I do swashbuckler 4, then swords Bard 4, (and after that maybe the two level dip in Warlock?) Or swords Bard then swashbuckler? I think going to level four and each class is important for the ASI / feats. Lastly, is there any advantage to doing 2 levels of bladesinger instead of Warlock? I'm really trying to keep everything as Charisma based as possible.
Start with one level of Rogue if you want an extra skill proficiency (and if nobody else in the group can pick locks), but you should head over to Sword Bard immediately thereafter because that's where the core of your vision for the character is at. Then power through with Sword Bard for at least five, maybe six levels. I say five or six because you're going to be heading to level four for the ASI, and once you're there level 5 Bard is a huge power boost just one level away, since it both unlocks 3rd level magic and moves inspiration to a short rest recharge while upping it to a d8, and once you're at level 5 extra attack is only one level away. You may find this character ends up more melee or more spellcaster than you intended, but either way going for levels 5 or 6 of Sword Bard make sense. Either you need better inspiration recharge for sword flourishes and extra attack, or you are mostly casting and Rogue levels just aren't going to stack up to more spells and spell slots. After that get the Rogue levels if you still want them.
Judicious use of the Sword Bard's mobile flourish can help compensate for putting off getting the easy disengages of cunning action and Swashbuckler, and getting it first makes it actually an interesting ability when you get it, rather than something superfluous for you.
On the issue of Bladesinger or Hexblade for an additional dip, I'd say Bladesinger if you've got the stats. You are a dual wielder, which has mediocre synergy with Hexblade, since you can only charisma wield one of the weapons. The multiclass spellslot upgrade and Arcane Recovery with Wizard are, on balance, probably more useful to you than the pact slots from one or two levels of Hexblade. Wizard gives more and better spells, even avoiding ones where a mediocre intelligence stat is relevant , whereas Eldritch Blast is really the only major spell draw of the Warlock list (albeit one that might be worth it). I don't count Hex as a draw, because you are someone who should be using your concentration for Bard spells by this point. Bladesong's additional movement combined with Sword Bard bonus movement when you make and attack and Swashbuckler means that you could enter combat, attack with action and bonus action, leave combat, and have 50 feet of movement to do it with.
That said, a Wizard dip requires a minimum of 13 intelligence, and I would argue that you need a minimum 14 to be able to prepare enough spells and get enough of a boost to bladesong in AC to make a bladesinger dip worthwhile. This is hard to manage for someone who already needs strong Charisma and Dexterity scores.
Note also that if dual wielding and playing by RAW either Bladesinger of Hexblade will require the Warcaster feat or stowing a weapon to cast spells with somatic components, and you'll need an arcane focus of component pouch for anything with a material component. Figure out exactly what spells you want and what your DM will enforce to cast them before deciding on either.
