I don't want to put words in your mouth, but please bear with me:
The primary argument against racial ASIs is that mechanical representation of the obvious physiological differences between Elves and Dwarves is "biological essentialism" and-- at the very least-- dangerously close to implying agreement with assertions of biological essentialism between different human ethnic groups. It's not really my main concern, but I do agree with the goal.
But if they're going to tell us that Elves being statistically more DEX than Dwarves is unacceptably racist...
... I am not going to sit here and tolerate them telling me that it's less offensive that the wealthy (Nobles and Artisans) are more INT than the poor (Urchins and Criminals).
And... if we are not going to address the social implications, if we're just going to talk about realism, the idea that a character's childhood and education make more of a difference in their literal physical biology than their literal physical phenotype strikes me as deliberately ignoring the obvious.