I think the real world diversity of settings is far higher than the published setting diversity because published settings have to aim for the consensus. I like how in 5e though, all but 4 of the races are listed as 'uncommon' and explicitly called out as not necessarily being in every setting.
In my experience on the boards, they probably could have added 'halfling' to that list. There are a lot of DMs and players that just love the little races, but there are about an equal number that don't want them in their setting. Neither halflings nor gnomes are available in my homebrew, and I'm far from unique in that.
To give you the idea of the diversity out there, the only races allowed as PCs in my game are humans, elves, dwarves, half-elf, goblin, hobgoblin, half-goblin, orine, idreth, sidhe, pixie, and changling. I guarantee that among DMs that don't allow every published PC race, you'll get a very large diversity of options, from the DMs that are 'only humans' to DMs that only allow a selection of homebrewed races (and maybe humans).
My real question is: why should we have a set of "always-on" races
at all? Why should humans always be present? I agree that there is reason for them to appear in
most settings, both from an appeal standpoint (Most Players Are Human!

) and for thematic breadth (historical campaigns, campaigns based on various fantasy literature, etc.). But why should they be
guaranteed? Why do we accept races being labelled "common" and "uncommon," rather than the frank admission that even "common" races are not always present?
(For an example of that: both the Elder Scrolls and the Tales of Phantasia universes
lack dwarves. In the latter they've gone extinct, but in the former there were never really "dwarves" as D&D understands the term--they were a branch of
elves, or rather "mer," who liked machines and lived in a mountainous region, but they were still just as tall and thin as any mer. On the flipside, WoW "dark" elves are good and nature-y, "high" elves are cruel and ambitious, and there are no halflings nor any real equivalent.)
Maybe I'm just being cantankerous here. It just bothers me that it seems like the D&D community as a whole swallows the idea of "common" (read: approved, universal) races and "uncommon" (read: count yourself lucky if your DM lets you play one!!!) races, seemingly with nary a question raised nor a lash batted.
I'm not expecting De Niro level acting. OTOH, I do expect to be able to tell from your portrayal of your character that the character isn't human. I don't think that's too much to ask.
My players generally expect me, as DM, to make the attempt to portray different NPC's differently from each other. I don't see why players should be exempt from that.
Why would other species
inherently behave so radically differently from us? Logic still works the same, math works more or less the same, there are still physical laws and limitations (albeit different ones). Different human cultures already prioritize different things anyway, and sometimes vastly different things. We've produced Rome, Mecca, Tenochtitlan, and Hollywood, all within a span of what, 2500 years? Why is it that other species
need to be so alien that a "Turing test analogue" could pick up on it?
I'm not saying it's wrong to
like it when people portray non-human species with differences from what is expected of humans, but I honestly fail to see how the difference of (say) a few idiomatic phrases, and thinking about things like "I have a tail, so it might have physical effects" or "I have hooves, so walking on hard surfaces makes noise" is any meaningful kind of "acting" (more than what
all RPers should do for characters that aren't siblings). These things are simple awareness of the physiology of the character, no different than "acting" a character that is blind or missing a limb; and the former things boil down to differences in culture, which you could
almost certainly find analogues for, if not in a single real culture, then in a piecemeal of several.