Now, if you start to use any advanced social combat mechanic (for example), then you are left with the following outcome- either you preserve player agency, in which case players can engage in it, but the enemy cannot ... or you violate the fundamental design ethos of D&D (full player control of the player's thoughts and feelings).*
Maybe not? We should note that "combat mechanics" are not a form of task resolution, they are a form of
conflict resolution.
In D&D, when attempting a task, the player gets to be pretty specific about what they want to accomplish - "I want to climb to the top of the cliff" or "I want to open the lock" are viable tasks. "I want to convince him that the King is actually a giant chicken" is also a task, and such social tasks are by convention asymmetric - many of us don't generally apply them to players, because of the agency-breaking that can entail.
But, everyone knows that in combat, things are messy, and they don't get to stipulate the detailed fate of everyone involved. While we don't like it when characters die, or retreat from a fight, we don't think of that as loss of agency, merely loss in this conflict. Why, then, in considering a possible D&D social combat, do we think in terms of task-type results that would involve breaking agency?
If we were to apply social combat - aka social conflict resolution - mechanics, the results should be like combat. The win/loss would not be about
agency, but about whether your social attempts to resolve an overall conflict succeed or fail. Loss in a social combat should be about not getting what you want, not about being specifically convinced that the King is a giant chicken.
So, an excellent example of a social conflict/combat folks might be familiar with is from LotR - the scene where Gandalf and Friends come into the Hall of Theoden. The conflict is actually between Gandalf and team vs Grima Wormtongue. Wormtongue loses, but not by being convinced to do what they party wants! Wormtongue, socially, is almost as dead as Boromir is physically, but neither lost agency.
As for the idea of making puzzle solving and tactical choices dependent on die rolls ... if I proposed that to any table I've run, I'd get run out of town.
Well, tactical choices also don't solve conflicts on their own. They are filtered through tasks - individual combat actions, that
DO call for die rolls.
As for puzzles, though, die rolls are hardly the only possible mechanical representation we can use.
Imagine the mechanic: Every character gets a number of
puzzle hints equal to their character's combined Int and Wis stat modifiers (minimum 1), this pool of hints refreshes on a long rest. Solving puzzles is then influenced by the mental stats, but not completely controlled by them either. It also allows collaboration between characters, as they collectively gather hints.