Like the humans actually are basically the same, and it is racist to say that one ethnicity is somehow seriously different that another. But if we cannot say that about fantasy races, they cannot exist because the bloody point is that they're different than humans (and each other.)
The bloody point is that "this is fictional and therefore doesn't matter" is fundamentally incorrect.
The fact that, indeed, my housecat and myself are radically different beings isn't the bloody point. The bloody point is that people are people, and we are very, very quick to use people badly, and we have responsibilities with respect to people that we don't have to the fiction.