If you're focusing on Booming Blade usage, I agree about not going past 4, though I wouldn't even suggest taking Thirsting Blade if that's the case.
If you're focusing on weapon attacks, going deeper in fighter gives you another extra attack, if you can get to level 11 with it. So, then, it...