the only spell i ever outright banned during play was one from the spell compendium called 'distract':
1 round per level, 1 creature per level, medium range, will save, no verbal casting component.
doesn't sound too bad, but anyone hit by it takes -4 to concentrate, search, spot and listen AND can only take a single move OR standard action per round. my player said it was reasonable, so i compared to slow, and explained that a 1st level spell that removes an action is pretty unbalanced. he didn't believe me, so i had a enemy caster use it and it nearly resulted in a TPK, cause the group's front-liners didn't realize they couldn't run, since they were locked in battle. also, they consistently failed to note the thieves sneaking up on them.
also, 'power word: pain' from races of the dragon.
people have yelled at me for saying that it's a broken spell, but please hear me out - it does damage based on your total hit points. it is close range and offers no save, but if you have less than 50 hit points, it lasts 4d4 rounds, doing a d6 per round. most characters under 5th level are going to have less than 50 hit points, so basically, it's just a really
way for the DM to say "Ok, you were hit with power word: pain. um...i rolled a 9, so you take 1d6 over the next 9 rounds." i realize this isn't an insta-death to most, but anyone under 3rd level just died, unless their cleric just so happens to have a full compliment of spells and potions.