To be fair, perhaps you're applying a higher standard to halflings that to other species? Like sure, I would combine them with gnomes, but then again I would also get rid at least half of the other lineages too. Most of them just exist because there were many words on thesaurus or because someone managed to think yet another marginally different and absurdly specific elf subspecies. Are orcs, goliaths and bugbears actually conceptually that distinct form each other? Do we need water genasi, triton and merfolk to be separate things? Leonin and tabaxi? And all the bloody elves?
A lot of the humanoid variants also exist because, until 3E and especially 5E, they were each tied to a narrow band of when they were a viable enemy.
Kobolds --> Goblins --> Orcs --> Hobgoblins --> Gnolls --> Bugbears --> Ogres --> Giants
The 1E PHB literally let fighters take as many swings on monsters below one hit die as they had levels, so that they could mow down, eventually, 20 in a round. Kobolds and goblins were intended as mass cannon fodder, unsuitable for even level 3 or 4 player characters, when they would have graduated on to orcs and hobgoblins. But by sixth and seventh levels, orcs and hogoblins were jokes, and it was on to gnolls and bugbears and ogres.
Since monsters didn't get levels and there was no notion of bounded accuracy, they had to create lots of different species to fight, even if there was very little conceptual room for them or if the differences were cultural or alignment-based and nothing else. There were something like three or four alternate "races" of orcs whose defining difference was not being Chaotic Evil.
And yes, sometimes Gygax (it was often Gygax) came across a new word or name for something that, realistically, already existed in the game, but felt the need to make it a brand new monster. And thus we got the xvart, the least loved of the 1E humanoid races. Basically, blue goblins who hang around with rats.