So, yes, there are roughly as many ways to handle this as there are systems in which to handle it. It seems plausible to me that any resolution system could land on the OP's outcome, the same way the application of 5E's did. What then?
Also, what does "yield" mean for an insulted tyrant? Specifically, an insult questioning his fitness to rule? From the OP, it sounds to me as though he did yield in part, allowing the PCs who didn't attack him to leave in peace. There's been a lot of ... presumption, I think, that the whole outcome was GM Fiat, and I don't believe that to be the case (other than deciding what the tyrant wouldn't give up, which seems like the GM doing the "playing the NPC" part of the job).
Not trying to answer for
@pemerton...I’m curious to hear his response. But I think this is a good question.
I don’t think that the situation in the OP was arrived at purely by GM fiat. I do think mechanics were deployed at times and that those helped shape the results. But it’s not entirely clear what mechanics, when, how often, and what results.
I also think that there were likely many points that were decided by GM fiat of some kind. And possibly some lack of clarity about possible consequences. It’s hard to say.
For instance, when the one PC insults the burgomaster and the burgomaster responds by calling “Guards!”, was any kind of check used? Did they DM simply decide “okay he’s not gonna tolerate that, he’s gonna call for his guards”. Additionally, when the burgomaster yelled “Guards!” did the DM offer any additional information to the players? Was it “Guards! Escort these ruffians from my hall”? Or was it “Guards! Kill these outlanders!”?
If it’s a case of no mechanics being deployed to determine the Burgomaster’s reaction, and then either an unclear threat (“Guards!” without any further cues) or an overt threat (“Guards, kill them!”) then I think that the DM has largely created the resulting situation by fiat. He decided how the BM reacted, he indicated a threat to the PCs, they responded.
Now, if that is the case, I don’t think that’s really a problem in and of itself. I’m sure many tables would consider all this well within expectations. But if this end result is dissatisfying in some way to the participants, which seems to be the case, then we need to look at the points where things may have gone differently.
So what if the one PCs insult was attached to an Intimidation check? The DM could set the DC for that and then call for a roll. On a success, maybe the BM doesn’t just start calling for the guards. Maybe he gets angry....but realizes these are capable outsiders, and perhaps he should try and keep a cool head. Maybe the insult actually gives a bonus to the other PCs’ attempts at negotiation. Maybe the DC is lower for their next check, or they gain Advantage on the roll.
On this way, maybe the bored PC feels he’s contributed in a meaningful way, and is a little less bored as a result. This seems to be one way to handle things that hasn’t even been considered in the discussion. A positive result to the insult.
Let’s say the Intimidation check fails. Maybe the burgomaster raises an eyebrow at the PC. Maybe some guards enter the room or advance in some other way...but the BM raises a hand for them to stop. “Mind your tongue, outlander, or I’ll have my men rip it out.”
This becomes a clear indication that things are about to escalate. It’s not vague. The PC can now press his approach and face the consequences, or he can back down and let the negotiations continue, or try some other approach. Alternatively, or additionally, maybe the insult makes the negotiation harder; the DC goes up or they get disadvantage on the next check.
I think very often the GM can get very attached to an idea of the “way things are”, and can become resistant to allowing change. I know this used to be true for me, especially with certain “darling” NPCs of mine. I’d be very reluctant to allow any input other than my own to affect them. Now, I don’t think the Burgomaster of Vallaki is anyone’s darling NPC. I think he exists as a foil to the PCs, but not an incredibly meaningful one. Allowing the PCs to influence him seems well within what we should expect from the game. He’s certainly not meant to be some insurmountable obstacle. By contrast, Count Strahd would be a NPC that I’d consider far more difficult to sway in such a way.
I mean, isn’t the whole point of playing to see how the PCs impact the world and how they are impacted by it? So I thibk it’s a good idea to either allow game mechanics for a chance at that, or to take it strongly into consideration when deciding anything by fiat.