You're asking a very good question for the future of the game. Why have them at all?
There are at least a few rock-solid fantasy RPGs that don't have "fantasy races" at all, where all of the PCs are either human or variations on humanity. I'm not even
advocating for that, so much as reminding people that it's possible and viable to have a fantasy game/setting that doesn't have the "standard" fantasy races. There are point-buy games where there's neither "class" nor "race"...
just points... and your character is whoever your points say they are.
There are very good D&D settings that deliberately omit some of those standard races, in favor of wild reimaginings of the ones left, and the introduction of newer, stranger races with less established lore. Those newer, stranger races... have settled, with some resistance, into being the "standard fantasy races" of newer editions of D&D.
"Why have them at all?"
I don't understand why anyone would want
all of the baggage that comes with a world populated by dozens of different kinds of sentient peoples... all of the traditional baggage and the stereotypical baggage and the
mechanical baggage... if they're not going to make use of those things. If being a "dwarf" or an "elf" is just a laundry list of stereotypes that aren't true of
the only dwarves and elves in the game, why are those labels even included? Why are they
necessary?
I use these things in my games, in a combination of "traditional" tropes and my own unique spin on them and... you know...
blank spaces for my players-- or
your players-- to fill. But I'm moving steadily in the opposite direction of where Official D&D (and/or mainstream fantasy) game design is going, going back to
hard limits and even race-as-class.
I think it depends on who you're designing the game for. From design space, you're giving an additional level of customization for characters, but you also have to deal with all the historical baggage associated with those different creatures.
I just struggle to understand how that's
undesirable, how that isn't part-- the majority of--
the appeal to having these non-human beings in your game and playing as one of them.
That's definitely the way the design is going, and if that sounds a bit incendiary or salty, I apologize for it.
No need to apologize... I agree with your assessment and
I get it. One of the advantages of being an apostate is that I can dislike the direction Official D&D and Mainstream D&D are going, and just keep on going the direction I was going in anyway.