we have few ways of thinking about this other than through the lens of Victorian-era, Euro-centric racial pseudoscience and centuries of this kind of cultural baggage
I mean, no, that's not actually true.
The problem is more complex and I'm sorry but America and American racists and racism specifically are actually a big part of it.
We have two really good lenses which are in fact much more useful and I'd argue kind of more common than weirdo racial psuedoscience. Science fiction and fantasy. Both used to be involved with the psuedoscience, but even by the 1960s were moving rapidly away from it, and we've had decades and decades of SF and fantasy which have much more interesting and relevant meditations on non-human species.
But there are three problems:
1) Racists in the US have worked incredibly hard to keep "racial pseudoscience"-style racism alive and part of the consciousness. They nearly lost the battle in the 1980s and 1990s, but thanks to the "brave" efforts of monsters like Charles Murray (author of The Bell Curve), they managed to keep it going, just barely, until 9/11 let the Alt-Right pick up the banner and use the internet and "ironic racism" (which rapidly curdled into "just racism") and then Facebook and then other social media to massively re-spread all that bollocks and remind your grandpa that weird racism was cool, actually. Plus there were just lot of Americans who had parents/grandparents with insane racist beliefs. RPGs weren't helped because Gary Gygax himself was absolutely racism-pilled (was a Kentucky Colonel from 1970s, "nits make lice", etc. etc.), and a number of early RPG designers were well, kooks with kooky racist beliefs (some still are, c.f. Tweet deciding to go hard on "scientific racism" for a while).
2) Science fiction gives us a much better lens that a whole bunch of people mindlessly reject because "science fiction isn't fantasy". Ironically many of those people do then fall into racist pseudoscience tropes, which are themselves essentially bad science fiction.
3) Written fantasy also has a lot of stuff, some really good here, but the reality is, about 80% of people who write RPGs have barely ever read fantasy beyond perhaps Tolkien and Jordan (or maybe Sanderson, today), and it's not any higher for the people who play it either. I do think this situation is improving, but slowly.
TLDR: It's not that we don't have other good lenses, we do, they just get rejected or ignored, whilst the racist pseudoscience lens is continually culturally promoted, and people fail to realize they're unconsciously using it, despite being the intellectual equivalent of the miasma theory of disease - wait what's that I'm being told...