I think it depends whether "optimizing a whip" means taking the whip as a given and trying to build the most effective character within that set of constraints, or whether it means "finding a niche where using a whip is better than other available weapon choices". The second one is I think the more interesting, so I'll go with that for this answer.
The main thing that makes a whip unique is that it's the only one-handed melee weapon which has reach. So we want to look for a niche where it's important to be using a one-handed melee weapon, and where we get a lot of mileage out of the reach property. It also has the finesse property, which may or may not be relevant. The main downside is that it's a d4, which makes me think we want a build that gets most of its damage from sources other than the weapon die (or at least one where the reach property is worth the tradeoff in damage).
Rogues don't care that much about the base damage die, but they have a few issues: (a) they don't get whip proficiency without multiclassing, (b) reach isn't worth that much to them, since they have cunning action disengage, and the main reasons to be in melee as a rogue are being able to TWF (the whip isn't light, so would need a feat), being able to use booming blade (which has a range of 5'), or maybe being able to use a shield (if they can get the proficiency, but even then it's questionable, since being at range is probably often worth more defensively than +2 AC). So to be a rogue that really takes advantage of the whip, we need at least one level in a class like fighter to even use a whip, and at least one feat to enable either booming blade at reach or, to use TWF with a non-light weapon (in which case the main impact of reach is avoiding AoOs, which the Swashbuckler can already do without any feats or multiclassing while dual-wielding shortswords).
The Wrathful Smite Paladin is a nice idea, though you're sacrificing damage on two attacks by using a d4, and the frightened/reach combo only works a few times a day, and then only if they fail their save. So I'm guessing you'd wind up wanting to swap to a different weapon some of the time.
Reach Sentinel lets you threaten more creatures with movement-halting AoOs, but you can still only make one reaction attack, so it seems like it'd require some fairly specific positioning to be a better option than simply closing in on one enemy with a more damaging weapon.
Taking Spell Sniper for reach booming blade is intriguing, but taking Mobile and using non-reach booming blade (with a higher damage weapon) accomplishes the same thing unless you really value the free cantrip that ignores partial cover too. Lore/Glamor bards are the first that come to mind when thinking about who could really benefit from booming blade and eldritch blast, but they don't get whip proficiency without multiclassing, and if multiclassing for martial weapons they may as well take a Hexblade level, at which point they get eldritch blast anyway. So then it feels like we're back to why not just take Mobile?
Hmm...