Taking "usually" at it's meaning is not over interpreting the word. If you read that as "most of the time, it'll be stealth, but sometimes it might be something else" is literal, not over interpretation. I don't have to do anything other than refer to the dictionary to read that sentence. This is not a strong argument -- you've set up a false binary where your side is that "usually" can be safely ignored as irrelevant but anyone that doesn't ignore "usually" must be over interpreting the word. There's a clear middle ground were you just take "usually" to mean usually.
You're missing my point - I hope not willfully. The rulebooks are meant to be read as rules - each word examined for how it's defined and what it means. The Sage Advice Compendium is an aide to help you understand and interpret the rules - they're not rules themselves, and they haven't been written to be read like rules. And, I haven't said it can be ignored. Have you watched the Sage Advice video yet with Jeremy Crawford where he talks specifically and a length about opponent distraction and alertness factoring into whether you can be hidden for surprise determination? It seems reasonable to me that in a video specifically talking about surprise that he would call out those edge case circumstances, and that's exactly what he calls out.
See, this is where it's clear that your bringing a previous edition's way of thinking into this game and analyzing it according to that set of assumption. Or, perhaps, a different game altogether, but this reads so close to 3e and (some) 4e thinking that it's pretty clear you aren't evaluating 5e according to 5e. For instance, the core play loop, on page 4 of the PHB, is that the GM describes the scene, the player announces PC actions, the GM determines if that action succeeds, fails, or is uncertain. If uncertain, the GM calls for whatever ability check they think matches the uncertainty, and the player rolls. The GM, in all cases, then narrates the results and the loop restarts.
You seem to be making some assumptions about my background being influenced by 3e/4e thinking. I played AD&D in high school and college (yes, I'm that old), and didn't return to D&D until 5e came out. I don't know if you're familiar with AD&D, but the rules were so scanty and unworkable as written that you practically were required to house rule a lot of things to make the game playable.
You seem pretty close here to arguing that there is no such thing as "rules as written" in 5e, but I think that's a fringe opinion if that's what you're arguing. My opinion, on the other hand, is that if you have a long-term group that have played together for years now, like I do, that if you DM enough you'll encounter these edge cases, and if you haven't worked out a consensus along the way about what the rules mean, you'll have continue to have tense moments in-game where players feel you're being unfair.
I'll give an example from early on in our play from a few years back. My closest friend of many years was playing a ranger who was parlaying from a distance with opponents in pre-combat role-play conversation, and suddenly says he's going to shoot an arrow at him. I ruled that initiative should be rolled and he came up like next to last in the order, and was very upset at the time. In fact, it wasn't until the Sage Advice Compendium came out that I feel he truly felt comfortable with that being how that situation should have been ran.
I'll give another example where it turned out I was wrong, also from a few years back. In a homebrew addition to Tyranny of Dragons, I had the party repelling down the side of a subterranean cliff-face in Undermountain, and there were caves with opponents along the way as they descended down. As they reached the first cave going down, I did a surprise check, some characters were surprised, and they took some damage before they could continue to descend. One of the monsters which was able to climb followed them out and continued to attack them as they descended. When they came to another cave opening with more opponents, I did another surprise check. After the session, one our players was very upset, and this is the one and only time I lost a player over a rules dispute. Maybe I'm scarred from it lol.
Anyway, again it took the Sage Advice Compendium to come out before I recognized I was wrong. To me, it was the same as if the party had gone from one room to another in a dungeon, regardless of the fact that the movement was vertical instead of horizontal. But the fact that that one monster followed them, and that they therefore remained in initiative order, meant that this was one combat session, not two. And by the rules as written, I shouldn't have done a second surprise determination.
Note, in this play loop, there's nothing that says, "unless we also print a specific rule in the book, it cannot be done." This is because 5e doesn't work like that -- it's not a game of prescribed rules. Instead, it provides useful, general rules to deal with things. This is why the surprise rules give a concrete example of how stealth works but leave it open to other options which aren't detailed -- it's because all of the ways you would do it are already enumerated on Page 4 of the PHB. Even the stealth example is just that loop, only dealing with a specific set of circumstances. Arguably, you don't even need that example bit of rules -- it's all still the play loop.
To me, you're continuing to argue that there's no such thing as rules as written. But, if there is no rules-as-written, the DMG wouldn't say you're free to ignore them. I think there's a very good argument to be made, one that I agree with, that exceptions should be made to surprise in support of character development and skill. I replied earlier about how I would incorporate that into the surprise rules if a character had infiltrated an organization that the party then meets in combat. But just because the rules intentionally give the DM leeway to flex them doesn't mean that there's no way to determine when the DM is doing that and when they're not.
If you really want to have 5e work well for you, you might want to step back and re-read the rules on how to play the game. Not because I think you're doing it wrong -- fun is the only right way to play anyway -- but because the way 5e is presented is very well crafted to enable that play loop, and lots of rules areas that appear grey or enumerated are actually quite simple if you follow the core guidance. I'd recommend Page 4 of the PHB, and the sections in the DMG on the use of dice. I'd recommend adopting the Middle path, as well, but that's preference -- it works very well for me.
You absolutely can decide surprise using social interaction rules. Just because you're stuck on stealth because that's the only thing that the rulebook explicitly outlined a play loop for you doesn't mean you can't get there other ways.
I do find some irony that a conversation spurring by our group stepping back to re-read the rules elicits a response that implies we're in dire need of doing so. Here's a quote from the DMG, mentioned in light of adapting the rules to a particular campaign setting (DMG p. 41):
"Flavorful descriptions of actions in the game don't change the nuts and bolts of the rules, but they make all the difference in the feel of the campaign. Similarly, a class doesn't need new rules to reflect a cultural influence; a new name can do the trick."
To me, that says this : the rules are nuts and bolts. When you're wanting to give character's a sense of satisfaction when they try something thematic but potentially game-breaking if it's abused or repeated, the designers envision that your first option is to follow the rules you always do, but narrate the results in-line with the character's attempt. When the character attempts to "surprise" via Charisma, you do the narrate that at the table as if there were no mechanic behind it at all : "Your bard approaches the group of bandits disguised as a another bandit, but one says 'Hey, boys, when was the last time one of you cleaned under your fingernails, eh? He's not one of us.' And you do all of that without calling for a single check.
Here's what the DMG says about Rules Discussions (p. 235) " "You might need to set a policy on rules discussions at the table. Some groups don't mind putting the game on hold while they has out different interpretations of a rule. Others prefer to the the DM make a call and continue with the action. If you gloss over a rules issue in play, make a note of it (a good task to delegate to a player) and return to the issue later."
Nothing there about making up your own rules on the fly being sanctioned by the rules-as-written unless you're house-ruling something, making a call to have the game continue uninterrupted, or trying something out with the consent of the players as an experiment.
Nothing in the surprise rules is exclusionary. It's why I pointed out that even changing "threat" to "foe" doesn't make a difference, it's only when you add "hidden" that causes problems.
You're making that up. Nothing in the rules gives any indication that surprise rule is meant only as guidance to the DM. It says the DM
determines surprise, and then in the next sentence tells the DM exactly how to do that calculation. The definition of "determine" that this clearly intends is this one: "ascertain or establish exactly, typically as a result of research or calculation." I don't add hidden, the rules do, and all of the interpretation help in Sage Advice presumes that surprise is being calculated as passive Perception vs Stealth. There's not a word in there that suggest it's presuming that the DM could be determining surprise any other way.
As for the social rules, I laid out that path -- the "ask" is to get the opponents off guard through social interaction. Presumably, they start at "hostile" so you have a long road ahead of you to get that to at least "indifferent." I don't know about you, but someone hostile to me isn't going to accept a friendly gesture from a foe that causes them to drop their guard -- they are hostile after all. So, any attempt to do so fails, no roll needed.
I'm on-board so far.
You'd have to improve relations, and do so while no one else in the situation does anything at all -- a case I'd certainly not allow happen because that's weird that your party pseudo-diplomancer can freeze everyone in place while they have a long chat. But, assuming that the PCs do get thing to indifferent, and then do make the "ask" roll to get the other side relaxed and off guard (at a minimum a group check, because everyone has to do the deception, here), then giving the party surprise seems quite valid. I mean, it would have been a heck of a lot easier and quicker to just hide and ambush the other group. I don't see your conjecture being anywhere near gamebreaking unless you include a lot of other non-RAW assumptions about how the game works, like a PC can declare a check to get the hostile other group off guard AND hide his friends. That's just not a doable thing unless you allow it, in which case the problem is, again, not with the surprise rules but with how you've chosen to run your game. And, if it's fun for you, it's not even a problem; but, you shouldn't need to rewrite the surprise rules, either, so I'm guessing this isn't fun for you.
Now, I'm no longer on board. I find it quite telling that nothing in the "Social Interaction" section of the DMG says anything about surprise. Nothing at all implying or stating that if you change an opponent's attitude from hostile or indifferent to friendly, that you can now suddenly attack and use the surprise mechanic to cause them to lose their turn. If that's what the designers intended, why not mention something so basic to the start of combat?
And if that's what the designers intended, after having defined these terms in the DMG, why not mention that in the surprise rule in the PHB? Say something like: "Any character or monster that doesn't notice a creature they deem as hostile or indifferent is surprised at the start of the encounter.". And something like "...the DM compares the Stealth checks of anyone hiding with the passive Perception score of any creatures deemed hostile or indifferent to them." But, that's not what the rule says, and I find nothing that implies it either in the PHB or in Sage Advice.
If I was DMing this situation, and PCs had convinced opponents they were friendly but then attack, I'd roll initiative as normal with there being no chance of surprise. If any monsters rolled a low initiative, I'd narrate their response as seeming very "surprised" on their turn. I might even have the previous change of attitude influence the opponents choice of actions on their round : "Since your foes were pretty convinced you guys were now friendly, monster X is very confused when you try to attack, and only tries to disarm you as they see your attack coming." I might even have them take the disengage action initially as they try to figure out what happened. What I wouldn't do is try to abuse the surprise mechanic.