I've done it A LOT.
Things I've learned: no matter how much you might think it doesn't matter, playing the following things is detrimental 9 out of 10 times:
- Music the players are very familiar with
- Music with lyrics (unless they are drowned out or very non-intrusive)
Aside from that, some of my best techniques have been coming up with themes. Not just for a significant character, but also for the campaign itself. So, I start nearly every session with the same CD of like 8-12 tracks, and this becomes like the campaign theme music. When that CD starts, after the first session or two, people just know "that's when the gaming begins."
But themes for NPCs are good, too. Especially if you do the cinematic entrance of the badguy and have appropriate theme music. Therion's "Enter Vril-Ya" (sp?) served this purpose better than any song I've ever used. My players shudder when that song plays, even if it's not at a game session!
Just as important: themes for the players. This may be harder to achieve and implement, but I try to find like 1-5 songs that are bad@$$ that I can play when things go the players way. It's nice to have themes for the villains, but if the Players know a song means that they are doing the butt-kicking, that empowers them. Then you can play with that later, and start the song, but cut it short or something and screw with them ;-)
My biggest problem is finding incidental music that doesn't put people to sleep. This took me a while to figure out: sometimes, no music is better than music that is slow and plodding. There are subtle effects at play, so you have to watch out. At the same time, if all you play is death metal every combat scene, that can get stale too (especially for players who hate that music). Most people don't talk about this with their players, but honestly, it's worth chatting about. If somebody loathes Gregorian Chants, then no matter how appropriate they are for the Church of Pelor, it's better to just leave the music off and concentrate on the game.