I am not entirely sure these all that separate. As we'll see in a moment...
So, with respect, etiquette is a set of rules. So, you were using rules, just awkward ones. We have to go one level down to see why you need any rules at all.
The broad analogy I'd use is... Cops and Robbers. When kids are all happy, and not terribly invested in any particular outcome, it goes smoothly - fingers get pointed, kids shout "Bang!" and everyone has a good time. So long as nobody cares, we don't need rules.
But eventually, someone actually cares about what the outcome is. Then the arguments start: I shot you! No you didn't! Yes I did! Yeah, well I'm wearing a bullet poof vest, and the bullet bounced off! Oh yeah, well the bullet bounces off the wall and hits you in the eye! Nuh uh, 'cuz I have bulletproof goggles...
Rules exist, in part, to arbitrate and prevent this. We agree upon the mechanism for resolving things, and we have our expectations set, and we eliminate a great many awkward moments. Rules are by no means the only way to do this, but they are a common and handy one.
Thus, when you want a game to be about X, and you tell us it is about X, the people picking it up are likely interested in X. And then, you can expect that folks will eventually care about the outcome of an instance of X. And it will become awkward if you don't have some established way to handle it.
When players enter into combat, they have some ideas of what the results might be. They know the rules, can judge risks, chances of success, and so on.
Your system... doesn't have that. The rules are not where the players can see. It is in the opaque system of the GM's judgement. There's no clear way to judge risks, chances of success, or know what the results might be - the player's ability to make informed decisions is extremely limited, unless they've learned how the GM's head works.
Also, if you are writing a system in which these social interactions are supposed to be a significant influence on how the game unfolds... do you want to leave that up to the individual GM? No. You hand the GM a system for those interactions, to help enforce the flavor of experience your game is intended to produce!