I don't believe that the WotC designers believed that the racial ASIs were meant to be archetypes at the outset of 5e. It strains credulity that they didn't assign racial stat bonuses as representative of biological abilities of the species they were portraying and were instead just trying to, paraphrasing "reflect archetypal bits of excellence in adventurers from D&D's past". We have a historical record here on ENWorld. I challenge anyone to go through the history of this board and find any quote anywhere from two years ago or before (maybe even before a year or so ago, I don't know when this idea started to percolate through Unearthed Arcana) that refers to the racial ASIs as anything but a biological representation. You won't find one, because it doesn't exist. It was cut out of whole cloth for the Tasha's book.
But are they lying? I don't know. They may have convinced themselves that they were thinking something different at the time, or they may have rationalized it in their head somehow, or maybe they just forgot and now remember it differently, or who knows what. I don't know. Of course it is possible, although extremely unlikely, that they DID intend it as that whole "archetypal bits" and just didn't tell any one for the past 7 years.
So, you don't want to say that they are lying. Maybe they have forgotten their design intention for a game that they were being paid to design and maintain. Or, as you say, maybe there is small chance they just didn't tell anyone.
Now, I'm going to look at this small chance, what would it mean? It would mean that when they said this was their intent, it was really their intent. That they told us the truth.
Now, let us look at the other possibilities in aggregate, all of them, what would they mean? It would mean that when they said this was their intent, they are lying. That they lied to us, maybe to prevent a seeming social Faux Paux.
And mechanically, or game design-wise going forward, what does this mean? Literally nothing. It matters about as much as the fact that Aloe Vera is succulent. Whether they are lying or not about their original intent changes nothing about the game going forward. And, additionally, you have come across something that is literally impossible to prove. You would need a time machine and the ability to read minds and follow them for the entire design process 24/7 to prove their actual real intent.
Believe they lied or believe they told the truth. The mechanics of the game don't change either way.
Yes, humans are more naturally charming and persuasive than elves. Slightly, like 2.5% better on average. And they are, on average, 2.5% less dexterous. And sure, Lotusden halflings can be wiser and more perceptive than gnomes. Why not? Isn't that more interesting than, "No, they are the same as everyone else." And why can't hobgoblins be smarter than dwarves? Illithids are smarter than dwarves, they are smarter than just about everyone! Why is it a problem with gnomes and hobgoblins but not Illithids? For that matter why is it okay for minotaurs and gnolls to be dumber but not orcs? It makes no sense.
Put those Racial ASIs back and nothing you said changes. Your dwarf still has his highest stat in intelligence because he is a battlefield commander and put his highest ability in intelligence and then put his class ASIs in intelligence as he leveled up. The fact that he has a +2 in Con and a +1 Wis or +2 Str doesn't change the fact that his highest stat can be Int. And he is, at worst, 5% behind a human that also focused on intelligence. Until the dwarf catches up and becomes the same. But that is just if you don't allow Tasha's optional rules. Which I would allow. So in my game you would not have a problem.
Your elf bard of 230 years has a high Charisma because he put his 15 there and then added another +2 at level 4, and another +2 at level 8. Is he as Charismatic as a Half-Elf or Human bard? Does it matter? Are you competing against the another bard in the party? How often does that +1 make a difference? Honestly probable a lot less than 5% since not everything you are doing is going to require a roll. Again, in my game I would allow the optional rule and it wouldn't be an issue.
Now I have to ask, are we just talking about PCs? Or are we including NPCs? Because if we are just talking about PCs, as I said, I'm fine with it. Your dwarf has a below average constitution for a "typical" dwarf because reasons, it can be anything you want. Be creative! Of course if your Con is a 12 or 14, you really don't need a reason at all, it is still well within typical dwarf constitution range. And the +2 Cha is easy enough to explain as "He spent all his time playing music and entertaining his friends when he was young." Fair enough! Sounds good! Or it could be as unusual as "was gifted with a fae-like glamour by a nymph after he protected her grove from rampaging goblins." Even better! Now I have a back story I can work with!
But if we remove default racial ASIs from NPCs, what does that say about dwarves in general? Nothing, it says nothing about them. "Bold and hardy, dwarves are know as skilled warriors, miners and workers of stone and metal." Really? Why are they known as such? With the new rules, if you remove all defaults, this is really not true anymore. "Their courage and endurance are also easily a match for any of the larger folk." Well, I guess? In that their courage and endurance are exactly the same. Why even mention it?
I guess the are slow (25' movement), have darkvision, are resistant to poison and have stonecunning. Or do they have stonecunning? Does it count as a proficiency that can be moved around? (Honest question here.)
Hill dwarves are a little tougher because of dwarven toughness, I guess. And mountain dwarves are... slightly better than everyone else at what ever it is they are trying to do? Like, they can an extra +2 ASI somewhere instead of just +1 and get an extra proficiency with any armor, weapon or tool? So they are not better at any one thing in particular, they are just better at... something. I just don't see how that is a better story.
See, but who cares what the NPC stats are?
Blacksmiths in my world's tend to have 16 strength if I bother giving them stats. Why? Because they need to be strong enough that the party is going to register "strong" when they do a thing. 12 isn't enough. Sure, it is stronger than average, but most of the party is going to have as good or better. So the blacksmith isn't strong at all.
Let us say that a Guard Captain is grilling the PCs and they need to lie to him. Decent sized town, I'd say they need to roll a 15 or better. If we reverse engineer that, that would give the Guard a 16 Wisdom to have their passive be a 15. And that is regardless of race. Dragonborn, Human, Dwarf, it doesn't matter, I'm not applying those racial modifiers to their statblock anyways.
And this is really the crux of the matter. Way back when for that "City-State of the Invincible Overlord" some game designer took the time to roll 3d6 in order for every single NPC. And someone else used a bell curve to show what an average stat was, then decided that a +2 to that stat would move the curve, therefore showing a slight increase in the average for a population. However, the truth is, very very very few people care, let only use that sort of method anymore.
You say that the statement, "Bold and hardy, dwarves are know as skilled warriors, miners and workers of stone and metal." is not true without that +2 Con and then a +1 Wisdom or a +2 Strength for hill dwarves and mountain dwarves, but that makes quite literally no sense. First of all, +1 Wisdom doesn't apply to any of being skilled warriors, miners, stonemasons or metalsmiths. +2 Con barely applies. So, what, are Hill dwarves not skilled warriors, miners, stonesmiths and metalsmiths? Of course they are. We portray them as such, we make sure that these facets are reflected in their societies, and we have NPCs say "Hey, if you want the best forged weapons, go talk to the dwarves, master smiths all of them."
And then... make that true. Stats literally never come into it. Especially since, if I need a dwarven warrior to showcase how tough and strong a dwarven warrior is... I'm going to be assigning him stats anyways. I'm not going to roll 3d6 and
hope that the slightly increased bell curve is going to give me what I want. I want him to be strong and use an axe, he has an axe and an 18 str.
Circling back up to your first paragraph though, your response was kind of "sure why not" which is a very strange response considering most people advocating for keeping these static ASIs as the default act like these things should have been blatantly obvious facts of the world. But, nothing about Lotusden Halflings tell us they are more perceptive except that +1.
You ask why a hobgoblin can't be smarter than a dwarf, because Illithids are smarter than dwarves. Well, reverse it. Why can't Dwarves be smarter than Hobgoblins, after all, Illithids are smarter than hobgoblins? When your example is equally valid in both directions, it becomes kind of obvious that it is mostly arbitrary.
Why is it okay for Minotaurs and Gnolls to be dumber but not orcs? Well, first that implies that they are being treated as dumber, which varies a lot. And second, if it is okay for Orcs to be dumber because Gnolls are dumber, then it should be okay for dwarves and halflings to be dumber because gnolls are dumber. Neither of those races get intelligence penalties either.
And again, this isn't about NPCs, all Illithids are part of a hive mind of hyper-intelligent psionic brains, being schemers and planners is part of their threat against the players. But, by that same token, so are Yuan-ti and Devils. And, high ranking leaders amongst fiends and Yuan-Ti also tend to have higher intelligence. Illithids more so, but, Illithids also use their intelligence stat in combat, so it needs to be higher for them to have the proper combat threat.
I mean, here is a fun little thing. Think about how smart you would play a Rakshasa. A cunning shapechanger and spellcaster who manipulates groups from the shadows, who is cr 13 and likely was the head of a criminal organization that the party has been tangling with for multiple levels as they slowly unravel decades of plotting. Think about how you would play that character.
Now go look at their intelligence. Did it match what you were thinking about this character doing?
Now take the other famous shapechanging fiend, the Succubus. The temptress who is CR 4, probably a minor side character for the real villain, or a low-level investigation threat. Do you think she is smarter than a CR 13 Rakshasa crime lord? Check.
NPC stats don't matter outside of combat. We will play them how we want, and change the stats if it comes to it.