Yeah, and this is exactly why I not only am not basing my preferences on the racism argument, but don't think it should even be invoked in the discussion of ASIs. Because once you start down that road, as you say, the same argument can be made about racial abilities. And I do love my racial special abilities! Just not racial ASIs.
That said, I do think that mental (Int, Wis, Cha) racial ASIs are especially problematic. It's fine to say that larger races should be stronger, but once we get into smarter, better looking, more charming, etc. things get...messy.