1) I don't think the base assumption is that they don't fit but that they fill a particular niche that would be missed without them. No one else in D&D are agrarian goodfolk who are worthy just for being decent folk. Elves and gnomes are fey, dwarves are mythic, humans are ambitious and heroic. Halflings are... human. Like that kind of human. They have all the potential of everyone else, they just aren't 'the special' inherently. They work for it.
But humans exist! And they do commoner and agrarian stuff too.
How are halflings more human than humans?
Now I am all confused. Are you saying that halflings exist just to have humans who can pretend that their D&D world isn't a D&D world? Isn't that outright saying they don't match the setting?
Okay, so NONE of those edgelord halflings are realistic. Humanist halflings, the ones with their shires seeking good food and comfort and family. Those ARE realistic. Modern Fantasy is so bloated and mutated and mutilated into something ugly and overbearing thanks in part to Martin but also dangerous amounts of 90's that we overlook that.
naughty words and cannibals and warmongers around every corner is not 'real'. Its the real of the pessimist and those who gave up.
They are more realistic as the halflings are changed to math the tone and realities of the new setting.
This is why I like 3e and 4e halflings. They still have that sense of optimistic and friendship but they heardstories of orcs invading baronies and gnolls killing/eating/sacrificing all the members of a town and dragons with a cult of kobolds. So they have organizations of warriors and push at least some halflings into diplomatic, economic, and militaristic fields.
I see joke halflings a lot. Mosting because many players, especially back in the day, have halflings be a lot more common than gnomes and half orcs as NPCs and many DMs/games were limited to the 4 core races.And again, I don't see halflings run as jokes as much as I do gnomes and half-orcs. How many non-big, dumb half orcs do you see?
Look at some of the old D&D clones. Look a some of the OSR games. Halflings art is often stupid looking. Halfling stats are often weakest. Halfling lore is often completely disconnected from the rest of the world.
But to a lot of D&D fans, halflings are sucky and silly. And they push the idea of silly weak halflings.Because to a lot of us... halflings aren't sucky. Or silly.
You can see it in alot of the discussions on TCOE and the new UA. "No. No floating racial ASIs. None of this freedom. You powergamers want to kill the lore. Halflings are supposed to be agile country bumpkins why can't fight and die in traps! You hire 4 halfling hirelings to enter the dungeon and you leave with one!"