My personal experience tells me different. It takes surprisingly little attacks to figure out a monster's likely AC, one guy misses, the next has just 3 more and its a hit. The DM doesn't need to reveal DCs or ACs for that to become obvious.
Pathfinder 2E is always about that little +1 that turned your attack into a hit (or crit). It works the same there, even though the game gives plenty of bonuses to all your rolls, with the ability modifier being rather small compared to to the rest.
It absolutely does matter, every little +1 counts, even in D&D 5. And don't make the mistake of thinking "but it's just 5 % difference on a d20 roll!". If you need a 10 to hit, but got a +1 bonus from somewhere, your hit rate might raise from 50 to 55 %, a 5 % absolute difference, but relatively, you hit 10 % more often. (This goes higher if the die you need to roll is less, and lower if the die you need to roll is more. For attacks, you often need less than a 10).
And the difference in modifiers in 4d6 ranges from -4 to +4, in point buy from -1 to +2.
I mean, I am not going to argue how you feel about it, but I can tell you, my feeling a +1 or +2 difference at the game table will matter, especially on stuff everyone in the party will roll often - which is attacks. For skill checks, often the GM won't require or allow the players to have multiple characters do roll on the same skill, so the second and third-best values won't come up anywhere as often.