The best advice I've ever heard for handling PvP (from Bawylie, maybe? Or Iserith?) was that when players want to PvP you don't roll dice: the target always narrates the outcome. If the target wants to roll their own dice before narrating, let them. This way there is no question about whether all participants are willing or not.
Andrew: "Ok, that's it, I cast Fireball at Bob's rogue."
Bob: "Uh...I dive out of the way, making my saving throw. Oh, look, Evasion. I take no damage."
EDIT: Just realized Iserith already posted in this thread with this advice. It works great. I had some high school students in a game recently and one of them was doing what I call "obstructionist roleplaying". When the other players killed the monster this one player had decided to try to befriend she claimed that what her character would probably do was go into a rage and attack them. I sprang Iserith's rule for PvP and the combat eventually fizzled out.
i have seen that guideline referenced quite a bit but to me, it seems to not do much more than shift the angle of attack or place of power.
ONE - At its core this is a player problem. there is a mismatch between the player's expectations as to what should and should not be going on. there is a conflict there that needs resolving. A game which allows the ambush style thing, the whisper to Gm thing, puts the power in the hand of the one who decides to take action. (Too much if there have been no long growing series of hints and clues IMO.) So, it may seem that this "target chooses" solves many of these issues. BUT in fact it means the player doing stuff the other player doesn't like has less reason to worry about or care about what the other player thinks. The possible threat of in-game resolution is now on the side of the player doing "bad stuff." So, if they want to just say screw it, they can keep going with these dark experiments and know that as soon as another PC starts to move to PVP they get to adjudicate the results.
So, if this is an attempt to let in-game mechanics resolve this issue, it doesn't succeed, it just hands the power to the other side of the player-v-player disagreement. As a general rule, we don't know which of them is in the right or in the wrong when writing the rule - so we still end up at an issue of one side in a conflict we have allowed to move from player-v-player to in-game resolution and allowing one side to have a basically unfair amount of power in that in-game resolution.
Now, of course one can say "dont make rules for bad faith" but we are dealing with a player-v-player problem anyway or at least a campaign coordination issue so in essence faith has already left the building.
Empowering the antagonizer with a "when attacked you decide what happens" sounds a lot like the oft-described case in sports where the player who "retaliates" gets the flag more often than not.
IMO at a wide variety of tables the more successful option will more often wind up being dealing with this out-of-character or allowing it to play out fairly in-character (if that has been previously established.) "fairly" in this case meaning a history of in-game chances for comment, observations etc of the growing discontent and after the fact in-game consequences to be played out. (i do not normally choose to use the latter as a default in my game but..."
However, one thing I tend to do in my games is this - an open statement that from the beginning "you may not be allowed back into the game with a new character if/when this one dies or leaves the party." I explain that if the reason for your character leaving is inter-party conflict and they kick you out or kill you or refuse to bring you back, that is going to prompt a big discussion about whether or not you come back as a player, is this the right fit, etc. Obviously that can occur at anytime, but it will be considered at these stages.
