I wouldn't go so far as to suggest it as a rule; it's just an approach to dealing with conflict. It's interesting that you see this as a requirement. I see it as the opposite. Freedom from a group or DM that requires the players to engage in role-playing and/or rolling dice when they don't feel comfortable with what's going on. I suggest a group allow players the ability to say "no, I'm not comfortable with that, let's work together and come up with something else."
Allow? How are you going to make someone play if they don't want to play? Tie them to their chair and withhold pizza until they agree to address the scene?
If it's not a table rule, but merely a suggestion, I'm not sure you've discovered anything novel. You 'co-authorship' approach is marked by what that is different exactly?
How would you apply this to the scenario mentioned above?
So the scenario plays something like this:
Bob: "I'm not going down there. There are dead things. The living do not belong with the dead."
Jim: "But Sir Regnar, we must save the Princess! You said yourself, the living do not belong with the dead. We must rescue her!"
Bob (who has simply wanted to create IC justification for acting out of character): "You are right Brother Tomas. I must face my fear for the Princess sake. But have your Holy Symbol ready! May the light of Showna protect us!"
Or
Bob: "I'm not going down there. There are dead things. The living do not belong with the dead."
Jim: "But Sir Regnar, we must save the Princess! You said yourself, the living do not belong with the dead. We must rescue her!"
Bob: "To Hades with the Princess. She's already dead anyway. What purpose is there in sacrificing our lives as well."
Jim: "Ok, if you feel that way. Stay here with the mules. The rest of us will go one without you, but don't expect a share of the treasure. You'll be missed."
Bob (thinking it over): "And leave me here in this graveyard alone. I think not! Wait for me!"
OR
Bob: "I'm not going down there. There are dead things. The living do not belong with the dead."
Jim: "But Sir Regnar, we must save the Princess! You said yourself, the living do not belong with the dead. We must rescue her!"
Bob: "To Hades with the Princess. She's already dead anyway. What purpose is there in sacrificing our lives as well."
Jim: "You know... you may just have a point. Why are we doing the King's business anyway? The Princess was always a vain twit anyway. Let's buy a ship and be pirates.
Bob: "Rrrr, matey."
OR
Bob: "I'm not going down there. There are dead things. The living do not belong with the dead."
Jim: "But Sir Regnar, we must save the Princess! You said yourself, the living do not belong with the dead. We must rescue her!"
Bob: "To Hades with the Princess. She's already dead anyway. What purpose is there in sacrificing our lives as well."
Jim: "Ok, if you feel that way. Stay here with the mules. The rest of us will go one without you, but don't expect a share of the treasure. You'll be missed."
Bob: "Ok."
Do you mean that the players who wish to enter the dungeon vote to kick the fighter PC out of the party?
I'm supposed to stop them? How?
What if that makes those players feel bad and now they can't look that other player in the face?
Then maybe they shouldn't have done it.
I think it puts them in a position of feeling like they're punishing another player in order to solve a conflict of interests.
Here you have a binary conflict of interests. Either the character will engage with what the rest of the party is doing, or he will choose not to. Either the party will decide to abandon the quest, or they won't. If 'Bob' chooses not to play, he can stay and watch everyone else play when they choose to go on without him. Perhaps he can run a henchmen with fewer qualms. I'm not going to make anyone play, nor for that matter can I actually make anyone play. But for that matter, just because 'Bob' insists on dividing the party doesn't mean I the GM can facilitate that by dividing my time equally between Bob and the rest of the players. Bob may actually be choosing to be bored, and there isn't much I can do about that.
If Sir Regnar feels punished, that's unreasonable considering it should be a relief to Sir Regnar to not have to go into the dungeon. If Bob feels punished, Bob is taking this too personally. If Bob doesn't like the fact that Sir Regnar's phobia of dead things is hampering his play, maybe Bob should play a character that isn't phobic of dead things.
Is it really better to use the game to punish people than to set the expectation that players (and the DM) can work out their differences?
Just because you 'allow' players to work things out OOC doesn't mean that they can or want to. I really meant it when I said I don't think you've discovered anything here. OOC communication can be useful for signaling to another player how you want a scene to play out or for when the player just can't play out the scene for some reason and would prefer to resolve it in a less graphic way, but it's pretty much never a solution to problems of incompatible desires. If anything, going OOC makes it worse, because you end up with various versions of: "If you want to be my friend, you'll do this." It's almost never a good idea to make an IC problem personal.