Could also consider Cleric (War or Tempest) 1 / Wizard X. Get the rapier proficiency, medium armor and shields same as starting fighter, but don't stunt your spell slots and can take Bless and Shield of Faith (obvs if there is a cleric in the party already this is less useful). You do lose out on the Fighting Style and CON saves though, which definitely hurts.
If I were switching to a full caster after Paladin 2 I don't think it'd be bard if you don't have access to college of swords. The bard spell list is fantastic but it's doesn't help your combat ability, so you'd just end up playing as a paladin with more smites. If you go draconic sorcerer you'd get the same CHA-dependence, smite slots and hit points as bard, plus shield, mirror image, misty step and haste. If you were willing to spend level 1 as just a caster instead of just a fighter you could even get CON saves.