Depending on level, I'd take a combination of the following:
-Front-line-combat types wacking at the monster in question, taking the penalty to hit for nonlethal damage. If any of them can trip or grapple well, that's a good option as well. If you're mid to high level, you might want to invest in weapons with the Merciful enhancement, which gives bonus damage but converts all damage from the weapon to nonlethal.
-Spellcasters 'spamming' the creature with their best non-lethal spells. Options include anything that weakens the creature (Enerviation), immobilizes it (Entanglement), or messes with its head (Charm). At lower levels, good choices include Ray of Enfeeblement, Sleep, Hold Person and Charm Person (for humanoids), Hold Animal (for animals), Bestow Curse, and Blindness/Deafness. At higher levels, Enerviation/Energy Drain can weaken monsters, Wall of ___ spells can trap them, improved Hold, Charm, and Dominate (as well as Power Word Stun) spells let you neutralize anything with a brain, and Polymorph spells can render the creature harmless (ditto for Flesh to Stone/Stone to Flesh). Reverse Gravity could be very useful (and a lot of fun).
-Rogues sneak attacking with saps, which deal nonlethal damage. They should then take a flanking position to keep getting sneak attack damage. Improved or Greater Invisibilty would help a lot, too.