Dimwhit said:
We're in the middle of a combat with Harpies. If a Harpy captivates you and you walk within 5', answer the following:
1. Are you considered helpless (one of the factors is 'being completely at their mercy)?
2. Could the Harpy do a coup-de-grace?
3. Is your Dex considered 0 or would you just lose your Dex bonus?
From SRD
Captivating Song (Su): The most insidious ability of the harpy is its song. When a harpy sings, all creatures (other than harpies) within a 300-foot spread must succeed on a DC 16 Will save or become captivated. This is a sonic mind-affecting charm effect. A creature that successfully saves cannot be affected again by the same harpy’s song for 24 hours. The save DC is Charisma-based.
A captivated victim walks toward the harpy, taking the most direct route available. If the path leads into a dangerous area (through flame, off a cliff, or the like), that creature gets a second saving throw. Captivated creatures can take no actions other than to defend themselves. (Thus, a fighter cannot run away or attack but
takes no defensive penalties.) A victim within 5 feet of the harpy stands there and offers no resistance to the monster’s attacks. The effect continues for as long as the harpy sings and for 1 round thereafter. A bard’s countersong ability allows the captivated creature to attempt a new Will save.
IMO the the target is merely prevented from acting. The ability does not state that you lose your dex bonus, are helpless, or suffer and other kind of penalty. It in fact states that the traget "takes no defensive penalties". Here is my take on your questions.
1. No, because the ability does not state that you are.
2, No, because you are not helpless.
3. Neither, I believe you would retain your dex bonus.
One could make something of the "offers no resistance to the monster’s attacks" part of the ability but in my opinion any penalties derived from that quote would have to be based on supposition.