I've played a few homebrew settings where some races were unavailable. As DM, what works best for me is to set out, from the start, which races and classes are available (and if unavailable, why).
In our 3e campaigns (a 'knights in shining armor' setting), the players got a list like this:
Common Races: Humans, Dwarves, Elves, Half-orcs, Muls.
Unavailable Races: Half-elves, Halflings
'Half-elves' are (more accurately) full humans or elves with a 'dual-heritage' backround. They are also PC only (no NPC half-elves). In this setting, Elves were best described by a line taken from the very first line of description of elves in the 3e PHB: "Elves mingle freely in human lands, always welcome yet never at home there." No half-elves (hopefully) reinforced this idea. Muls replace half-elves (in theory).
Gnomes were an uncommon race (less widespread than the others). Other uncommon races included goliaths, raptorians, and warforged. Illumians were unavailable due to the 'constructed' nature of their magical language (which was at odds with the nature of arcane magic in the setting)
Slightly 'narrowing' the list of available races was (hopefully) a way to subtly increase the prominence of humans in the setting.
= = =
Core Classes: Barbarian, Cleric, Druid, Fighter, Rogue, Sorcerer.
Bards, Paladins, and Rangers were prestige classes (as seen in 3e Unearthed Arcana).
There were no wizards. In this setting, arcane magic was sorta like the Force (from Star Wars); spellcasters learned to 'sense and control' arcane energy, rather than 'study and learn' (Intellect had no effect on magic, only Charisma).
There were no monks (although I would've been open to the idea of a character originating from somewhere far away). This was a land of mounted knights and crusading clerics but it wasn't the only realm out there.
There was also a list of 'expanded classes': Hexblades, Knights, Scouts, Spirit Shamans, Swashbucklers, and Warlocks. Available if anyone was interested.
= = =
Our 4e homebrew did away with 'traditional' fantasy races altogether. No dwarves, elves, half-elves, half-orcs, or halflings.
In this campaign, we made a definite nod to the races of 4e. We had humans (o' course), changelings, dray (a.k.a. dragonborn), devas, genasi, shifters, tieflings, minotuars, and wilden.
Devas figured prominently in the lore of this setting; a deva emperor ruled a 'Union' of worlds (actually mini-settings), traveled to via a series of magical Runegates. A fantasy equivalent (of sorts) of the TV show SG-1.
All classes were available (although we weren't using psionics).
= = =
As a sorta counterpoint to those games, we later played a campaign using D&D Essentials exclusively.
We had Humans, Dwarves, Eladrin, Elves, and Halflings. Drow, Half-elves, and Half-orcs, (and later) Hobgoblins were uncommon races (less widespread than the others).
The games were a nod to 'old-school' D&D. Dragonborn, tieflings, and warlocks were outta place.
= = =
I think having a clear idea of what these campaigns were gonna be about helped. Everyone knew what was available and why. The settings could still be distinctive (no need for the 'whole kitchen sink') and still have plenty of choices.
It also seemed to help us to have a list of books on the table (as it were). In our later 4e campaign for instance, we knew that we would be using D&D Essential rules exclusively.