By the time you hit CR4+ your fighters will be able to hit 4 times or more with Action Surge.
The -5 to hit isn't a big deal with so many ways to get advantage, plus potentially having Bless, a War Cleric or a Valor Bard to pump up your attack roll.
So yeah, it is a problem when you combine...