We're just ignoring the spartan example aren't we?
At any rate, is the thing you're suggesting that we can't write lore that's not based firmly in the rules? If all races are roughly neutral on a class-by-class basis, the world is liable to make more sense, because how many people become wizards would be based more firmly in the setting, rather than the mechanics of racial ASIs.
"It doesn't make sense for the elves to believe that orcs suck at magic."
Well, sure it does. It's just up to the GM to provide an explanation. Perhaps they've just never seen an orc wizard. Perhaps the orc tribes rely on oral tradition which made the development of magical schools difficult. Maybe the high elf education system is deeply racist and based on the beliefs of their failing empire. All of these are better justifications for these beliefs than ASIs.
It would be weird to say that we shouldn't believe the Amazons were great warriors because they don't have the right ASIs. The belief that the Amazons are made up of tall, powerful, warrior women is a cultural touchstone, not an immutable fact of life. Likewise for the belief that orcs are dumb.