I think the -2 to hit may work to balance on the other classes as well. The penalty to hit will lessen the boost from Hunter's Mark (+3.5 per hit) and from improved divine smite (+4.5 per hit). They also only get extra attack once, so we're only talking about 1 extra attack. It also balances twfing and gwfing at the low levels, where twfing is ahead.
Ranger (Hunter's Mark) would be:
Duelist: 1d8+7+1d6 (15), x2: 30, x 0.65 = 19.5
Twfing: 1d6+5+1d6 (12), x4: 48, x 0.55 = 26.4, a gain of 7 but the duelist has +2 AC. Colossus slayer skews this, but I'm in bed on my phone and I don't want to figure out chance of at least one hit on 4 attacks at 55% to hit right now ...
Paladin (11th level)
GWFer: 8.33+5+1d8 (17.83) x2 = 35.7; x0.65 = 23.2
Twfing (takes Defensive): 1d6+5+1d8 (13) and 1d6+1d8 (8) x2= 42; x0.55 = 23.1 (and has +1 AC); twfer has more chances to proc smite, but also has lower chance to hit each hit. Since total number of smiles is still limited, they'd have a higher nova potential. Also, their damage would go up if they dipped fighter 1 for twfing.
So, not a perfect fix, but not as bad as it could be.
I'm just spitballing here.