I still don't get this "feeling like a jerk" bit if I limit things in a specific game.
If you have been playing with people for a while, or they themselves have been playing for a while, then they should know that everything isn't always allowed. New players shouldn't be a problem either because I seriously doubt they would understand the whole "core" and "not core" aspect.
If you step forward and say: "okay lads, I would like to run X game but I am only allowing ABC books. Anyone up for it?" I simply don't believe in that player who is going to be crushed if he can't play his mutant ninja salamander, and if he is then that's life; just wait for the next game.
I totally know what you mean but I've also played in games where the DM laid down the law, then caved in when their partner said "I want to run a Half-Shadow Giant Hybrid Barbarian/Rogue/Psion, you're OK with that right?".
what about when what you want to disallow isn't weird or out there?!?!?
lets try this hypothetical
DM designs a basic 3.5 D&D world... says "Hey lets play, here is the world"
Player A says... "Cool, I want to play a half drow Warlock... my mom was a bar maid and my father a raider, I don't know where my powers come from, but I suspect my dad sold his soul to loth or something."
DM "No too weird..."
Player B says "I want to play a human Druid"
DM "OK"
Player A then says "OK, I'll play a Sword sage who travel's the world looking for new fighting styles"
DM "No, book of 9 swords is way broken"
Player B "My third level feat is the one that lets me cast in wild shape"
DM "OK"
Player A "BUT Sword sage is over powered?!?!"
DM "Just play something normal"
Player C "I want to be an Elven Ranger"
DM "OK"
Player A again "Fine, I'll play a sorcerer/monk with vow of poverty"
DM "No, I don't like that either"
Player A "So what can I play?"