I’m not sure what the dichotomy is, RPGs are shared Storytelling, not improv theatre.
I love freeform play too, I encourage it, but that doesnt mean I get upset by rules (though I do prefer narrative focussed games like FATE). It is fully possible to roleplay during combat, actively narrating your interactions and doing in character dialogue, as much as it is possible to roleplay in any other aspect of the game, the DM can still mediate, and dice as a randomiser provides a prompt for further roleplaying, it doesn't prevent it.
I mean, not to drag out the emergent narrative vs. plotted narrative horse, but improv theatre is absolutely a part of a lot of shared storytelling at a lot of D&D tables, and would be missed very deeply if it were to vanish. Which isn't to say there can't be rules - improv works well with simple rules that allow good riffing. It's just that prompts like random tables and personality traits tend to work better than a usage of resources to achieve a victory.
One thing that I posit that a lot of games miss about the appeal of D&D is that a more freeform social pillar scratches a different itch than the more dice and math heavy combat pillar, and that this is a
virtue, not a gap that detailed mechanics should step into. It works different brain parts, it brings different players to the fore, it works as a pacing mechanism and as a method of play that is more open-ended and less constrained than the mechanics-heavy combat. If you weave a lot of mechanics in there, you weaken that appeal and produce a more homogenous, less varied play experience.
Which isn't to say that social mechanics need to be absent or that certain classes or archetypes couldn't plug into some mechanics and benefit from that. One of my gripes about D&D for the last few editions is that everyone gets cool combat abilities in their class, but relatively few cool things to do outside of combat (even spellcasters are constrained on the social pillar, 'cuz no one trusts the dude casting spells at the party). 5e does a bit better than 3e/4e in this regard, but only a bit, really.
But the shape of social mechanics needs to look different than the shape of combat mechanics. It's not about expending resources to achieve a victory, it's more about introducing character traits and personal props and then just kind of seeing what happens when they all bounce together. More like playing with dolls. Describe what you're wearing. Describe how you sound. Even if your character is BAD at the social mechanics, the fun of play is still in describing how you are bad and what happens because you're bad. If I play a real rude boy, then farting in the Chancellor's face is a successful moment of social interaction, even if it means the Chancellor refuses to speak to us, because it means I impacted the story in a powerful way. Heck, the
player agency in D&D's traditional social play is huge! Bigger than in any well-balanced tactical fight!
This is part of why one of my favorite social mechanics that 5e brought to D&D is that playing your Flaw could (should, IMO!) grant you a free Advantage. Be an idiot, make interesting problems, have a cookie for it.
Bad social mechanics look more like...Chaotic Stupid, or Lawful Jerk, or Kender stealing from party members, or paladins in the party fighting thieves in the party, or "girlfriend classes", or thinly veiled racism given a fantasy coat of paint, or Wandering Harlot tables. Those are the kinds of things that have historically broken the social pillar in D&D, because they can turn
playing your character into a bad experience.