Well, in the real world you are absolutely right. Unfortunately, D&D is a game. At some point you have to realize that the "normal" poeple of a race have 10s everywhere and a 11 and 12 in two stats. Having the standard array makes a character quite exceptional already. From a +0/1 ability bonus to a total +5/7 is quite a step in the exceptional don't you think? This is a 500% to 700% increase.The world sees an atypical being it does not see where this came from nor care that they might be slightly more atypical than another atypical one of a different lineage..... ie much ado about simulation in a non-simulation
Even the 4d6 method makes for quite exceptional individuals. At some point, the goal of the game is about heroes struggling to make better world or simply to save it. At which point do heroes become super heroes?
One other thing to consider, the stronger the characters, the stronger the challenge they will have to face. And this will lead to a much more swinging game where the scales will easily tips in either a surprise TPK or becomes simply too easy to offer a real challenge if the DM does nothing to adjust the challenges.
The game is balanced as is for the tier 2 to 3 but becomes trickier the higher you get. Fixed ASI gives character advantages and balance these with disadvantages on various classes. No one argues that wearing medium armor is quite a step for a wizard. This why dwarven a wizard starts with a maximum of 15 in intelligence. With floating ASI this balance is thrown to the four winds and nothing prevents your dwarven wizard from taking it a step further and take heavily armored as a feat on level 4. With fixed ASI, it would be a major throwback as it would mean being behind by two points. A 5% sacrifice is not that big of deal. But a 10% one is.
But maybe I have been a DM for so long that a players' perspective is becoming an alien thing to me. On the other hand, almost all my current players share my view on this...