Your problem is one of misperception, I think.
In a "light" game, based on one-off modules among friends who do not put much emphasis on the campaign world, the plethora of monsters graded by CR and HD makes perfect sense. If most games are "pickup" games, this arrangement is just what you need to quickly throw together an interesting-but-balanced encounter.
On the other hand, in a game where the campaign world has more meaning, and there is an emphasis on setting a flavor and tone, then the key is selectivity. The various monster books give you the tools, but just as a painter does not try to use every shade of every color in a single portrait, so must the DM shape flavor as much by that IS NOT in his/her world as by what IS.
There are no halflings in my world; they do not fit the mood/tone I am building. The Goblins are called the Gar-Ishtrakyn, and inhabit a range of haunted hills along the northern border of the main human kingdom. The Orcs are called the Tarrakyn, and they inhabit two mountain ranges... one along the eastern edges of the main human kingdom, and other far north and west (where the Tarrakyn have slowly taken Dwarfhold after Dwarfhold over the centuries). Gnolls, called Canakyn, are from a jungle-continent away to the south. They only appear in this area when brought as mercenary guards by traders from the south.
The number of different sentient races in the MM alone is enough to make it highly unlikely more than a few of them come from the same area. The keys are geographic dispersal and selective use.
That being said, WotC is not entirely behind the curve on this. The 2nd Edition Birthright setting presented Goblins as one of the major races, without Orcs about (there were Orogs, but they had a different origin from the half-Orc/half-Ogre version). The Goblins had a kingdom that was on a par with the human nations in many ways. That was one of the best presentations of Goblins I have seen to date.
Also, check old Dungeon and Dragon magazines. I dimly recall an old (pre #100) Dragon magazine with some nice details on Goblin culture in a campaign world. Also, there was an adventure in Dungeon, back in the 2nd Edition days, that presented Goblins with PC classes and levels (well before the Complete Humanoids Handbook).