Maybe the title of this thread should be changed from "why do we have halflings and gnomes" to "Why do halflings suck?" That would fit the current line of discussion much better than the current one.
Halflings are treated different from every other race in the game. Even the "weird" ones, like Locathah, Grung, Aarakocra, Tortles, and Kenku. Every other race gets in depth description about their place in the world and how they survive. Locathah use weapons and pets, Grung use their poison and design their bases to keep out enemies, Aarakocra fly away, use weapons, summon air elementals, and live atop cliffs, Tortles wander around and protect their eggs in big communal settlements, Kenku hide in urban areas and do light banditry work.
However, Halflings get none of that. They don't even have the Tolkien explanation for how they live, Gnomes get the whole "small burrows under ground" thing. Halflings in the core game are more like a walking meme, the coalescence of every hobbit thief/bard stereotype in existence, refusing to satisfyingly answer the most major questions that their existence raises.
How do their villages protect themselves? IDK, they figure out to hide them all, I guess.
Why don't they want riches, land, or conquest?

Because reasons.
Why are they lucky? A god did it. What about in other worlds, where there aren't definitive gods (Eberron, Dark Sun, etc)? IDK, you figure it out.
Why are they brave?

Because reasons.
I could go on.
Halflings are underdeveloped and are lazy meme characters. I don't hate them, I honestly love Exandria's Lotusden Halflings, and the Halflings of Eberron and Dark Sun, the "core" ones are just uninspiring and lazily designed. Even their name is lazy, just ripped from Tolkien in a way that they couldn't be sued over.
Can we please stop with everyone saying that if we think halflings are lazily made that we hate Tolkien or D&D or whatever other nonsense people are professing?