The DMG recommends to never ever ever use XP penalties. I think this is a good recommendation.
If there are problems, talk to your players about them. If the problems persist, find other ways to penalize them, IC. Otherwise, stop playing with them.
Depends on how you award XP. Ways to distribute XP that I can think of as follows.
Combat XP should not be reduced unless you are especially picky about metagaming in combat or you have a player who actually LOOKS UP the monster they are facing during the fight.
Roleplaying XP should only be docked to people who consistantly play against how they say their character is. Examples would include someone who describes their character as a one who tries to talk their way out of trouble, but constantly uses violence as the first option.
Quest-based XP should only be docked if they actually fail to fufill the quest.
I don't use XP penalties of any sort - I never take it away (unless a character is level-drained permanently, or they "spend" it on magic item creation, etc.) and I never grant less than I would to try to influence player behavior.
I tend to try to use XP rewards for good behavior, rather than XP penalties for bad behavior.
In my game there are three classes of offense punishable by TXL (Total XP Loss):
backtalk
refusal to buy alcohol for the group
refusal to pay for the GM's share of the pizza
In addition, complaining that the GM is constantly throwing excess d20s at you is punishable by PXL (Partial XP Loss, ranges from 15% to 99% of total possible reward) and no alcohol.
Showing up to the game with no food for the GM, and no alcohol, is punishable by death.