I think it can be useful to talk about the micro-moments that lead to that jerk mercenary being a problem.
Like, you mention a sharp veer toward hostility, but in the right circumstances, that's not necessarily a problem. It's just a player saying, "I want to take the story in this direction." If everybody else is on board, it's not a problem.
The actual problem, in specific, is "interferes with another player's fun." A sharp veer toward hostility is just one situation that could, in the wrong circumstances, interfere with other players' fun. There's uncountable other situations that could do the same thing - whenever a player wants to take the story in a direction that's not fun for anyone else. Heck, even an extended scouting mission or a long social scene could fall into this camp as the grumpy dwarf in plate mail twiddles her thumbs for a few hours.
And that's not a problem with jerk mercenaries, that's a problem of
player skill at performance (and maybe a bit with DM skill at player management). And that's something we can put mechanics in to help with.
Which leads to a few design goals:
- It is more important to play as a group than to "play your character."
- Defining the goals of the PARTY, and getting the PC's aligned with those goals, is important. Similar to how you should play a D&D character who is up for an adventure, you should play a D&D character who is up for some collaboration on a goal. That goal might be personal (acquire a lot of money), or not (save the world), but collaboration with others is going to be an important part of how you get to that goal, even if your character is typically a lone wolf sort.
- The improv rule of "say yes" means, in part, don't negate someone else's contribution. That extends to players as well as DMs, who should not negate the offerings of other players.
So, collaboration is important, but there's no real mechanics in D&D today that encourage that kind of social collaboration. There could be! Imagine if every player got Inspiration to award to each other. Or if maybe playing your flaw granted the rest of the party inspiration (but not you!). Or if we have a "talking stick" mechanic that allows one player the ability to speak for what the party does, but that stick rotates each time the speaker fails a check.
If you play a jerk lone wolf mercenary but agree as a player to collaboration, then your jerk lone wolf mercenary becomes less of a problem child and more like...early-game Cloud Strife. As a player, you know your performance is given certain constraints - you won't negate other players' contributions, you will allow party to trump individual, you will be on board with the party goals. You can still perform your character as an isolative jerk, she's just an isolative jerk who is drawn - perhaps despite her best efforts - into being a team player ("It's not like I like them or anything...d...dummy."). Drawing your sword and threatening to kill everyone just becomes...your version of an Intimidation check. Not something that takes command of the whole scene.
And then this extends to other problem player types as well. The classic Klepto Kender isn't necessarily an invalid archetype, it's just a potentially antisocial one, like our jerk merc. If we can get players to prioritize the party and not negate each other, then the annoying "No, Jarod, your character can't just STEAL my vorpal sword" becomes the more collaborative
DM: "Okay, Noelle's turn."
Jarod: "As Noelle's character goes to attack, she realizes her belt is light. She looks over to my character and sees him absentmindedly drawing pictures in the dust with her vorpal sword. He throws it back to her with sudden panic as the moment sinks in for him. He says, 'I'll finish my portrait of you later! Get those dragon army soldiers!' Also, you can have an inspiration because I used my flaw on you."
Noelle: "My character says, 'Remind me to get a bell for you when this is all over!' and leaps to the attack!"
And much of this is pretty obvious for veterans, but the PHB contains precious little on this kind of play. It's assumed, practiced, but not codified, and not mechanized, so it's lost on a lot of players with otherwise great intentions.
If we want better social gameplay, I think this is the path we should walk down. Don't give me points and die rolls. It ain't about that. Give me ways to participate in a group performance, ways to make my flaws fun, ways to keep the group empowering each other, ways to be a better player of a role in the context of this role playing game.
Though D&D has done
something in this regard, it's largely been accidental or incidental, and I think that if we had some good design goals for this pillar, we could get some mechanics that aren't just "combat, but using different stats."