I mean, by itself, it's not really that bad. Sure AC doesn't rise much, thanks to bounded accuracy, but I look at it like this.
Say you're 4th level. You can fire an arrow for d8+3, an average of 7.5 damage. Giving yourself -5 to hit for 17.5 damage sure sounds great, but it's basically turning your attack bonus to +0! Well, +2 if they don't have any cover. That's still pretty rough.
Even next level, ok, you get another +1 to hit, and kind of sort of advantage with two attacks. If one hits, 17.5 is a boost, but you could just attack twice for 15.
The problem comes in when players go out of their way to find extra accuracy. Maybe you were a Kobold (RIP Pack Tactics). Or a Battlemaster and you used your superiority dice for an accuracy buff (not as bad, since you could just chuck them at the enemy in the form of damage).
But if your party is working with you? Mastermind Rogues, Wolf Barbarians, Clerics with bless (let alone Peace Clerics), Bards...there's lots of ways to give out advantage or numerical bonuses, and now our archer is suddenly doing 35 damage a turn!
But that does take investment, so I'm not sure that you can say that's broken...it's just smart play/good teamwork.
I never bothered myself, though. I hate missing. Worst thing that can happen to a martial character. "Ok, your turn is up, what do you do?" "I swing my sword and...oh."
"See you in 10 minutes when everyone else's turns are done!"