That's not actually true, though, because there's a floor, and that floor means that there's a window where the -5 doesn't represent an actual 5 point spread on the d20 where you miss.
Take a level 8 archer, for instance. Assuming only one ASI, that's an 18 dex, +3 proficiency, archery style, and a +1 weapon for giggles. That's a +10 bonus to attack. This means that the floor for this character is AC 12. Let me explain a moment what I mean by floor. The floor is the AC where no bonus helps, because it's unnecessary. Since the character always misses on a 1, you take the 2 that doesn't miss, and that plus the attack bonus is AC 12. No bless or extra bonus will improve the chance to hit AC 12 or lower - the miss chance, even if you roll a 4 on your bless dice and a 12 on your bard's loaned die (he's level 20, very helpful) will always be 5% -- the chance you roll a 1.
So, then, let's look at this from what this character picks for their ASI at 8th. On the one hand, you can go for a 20 DEX. That pushes the floor to AC 13. Or you can pick up Sharpshooter and your floor stays at 12. If you use the -5/+10, your floor drops to 7, which isn't interesting because only a handful of monsters have a 7 AC or worse. So, at this point, it's absolutely true that you have 6 points less spread on the d20 to hit an AC 13 (the -5 and the +1 from the loss of increased DEX). But it's -5 at AC 12, -4 at AC 11, -3 at AC 10, etc., so that doesn't hold universally -- there are ACs at which the -5 doesn't hold to a strict 5 point spread. But wait, there's more! For every point of bonus, that spread shrinks for the sharpshooter but doesn't for the DEX 20 shooter for all ACs less than 14. A 4 rolled on a bless ups the floor AC for the sharpshooter to 11, but doesn't improve the chances of the non-sharpshooter to hit the same AC. So, there are a range of ACs for which the sharpshooter gains full benefit from additional bonuses that the non-sharpshooter does not.
Now, if you climb up into high ACs, above the floor value for the non-sharpshooter, this changes of course, because then the bonus does help the non-sharpshooter reduce their chances to miss. But, for low ACs, the effect of a buff is all for the sharpshooter/GWM and doesn't aid the non-SS/GWM at all. This is the way that the penalty can be min/maxed away.
And, at 12th, the sharpshooter can take their next ASI to DEX and even it up. Honestly, the benefit is better at lower levels where ACs remain reliably low, so the 4 - 8 level zone is a bit of a sweetspot.