This is a valid question, and one that probably involves a little more communication and nuance than just a quick rundown, which is why I'm breaking this one question out into its own response. I've mentioned my answer to this in threads past, but don't mind talking about it again.
Going into it... the first question I would ask was how familiar you were with performance improvisation? Knowing us EN Worlders as we are... my guess is that you and probably most of us have at a minimum seen a few episodes of Whose Line Is It Anyway? and thus will recognize what I'm going to talk about (as they involve relatively standard improv jargon).
So with that being said... you are correct that I myself would not "need" game mechanics per se. I could (and have) completely improvised games and scenes and such-- usually on stage, but even at the game table (as something like Fiasco has the barest hint of game mechanics once the premise, characters, and relationships have been established, with the rest is primarily just freeform improv.) Obviously D&D is not that... as dice and rolling dice are a huge part of the game, as are concrete rules for adjudicating action. Now why would I play a game like D&D that indeed has and uses a lot of game mechanics when I don't really "need" them (or indeed find being beholden to them rather irritating at times?)
- Part of it is that it's the universal game system that everyone knows and thus is easier to get players for (just like everyone else finds to be true.)
- Part of it is that the "combat board game" part of D&D can be fun in its own right and I enjoy playing it. But I don't find the combat board game to be so great of a game that I need it to be the best board game there is-- and thus I don't get bent out of shape when any warts in the game show up. If some rules don't work and I need to make Rulings for them on occasion, whatever, that's fine, doesn't bother to me.
- And finally... part of using the "board game" of D&D is that the dice rolls take the place of an important and necessary part of scenework improvisation. When two actors are improvising a scene, in order to push a scene's story forward the actors have to build upon what has been established. The 'And' part of 'Yes, And'. The first actor makes a statement of fact within a scene, the second actor accepts this statement of fact as an agreed-upon truth of what is happening in this made-up scene... and then adds details to enlarge it, make the scene fuller, and push the story forward. The first actor then responds honestly to these additional details as being factual to this scene and then they too add more details. And that's how the scene will progress-- each actor accepting the truth of the other's statements and adding more truths on top of it, back and forth and back and forth until the scene reaches a conclusion (as determined by whatever format the improv is taking place in.)
But now in D&D... we have something extra. We no longer have to rely on just the two scene partners to add additional details to the facts of what is happening... we have a Dungeon Master and even more importantly, dice rolls to add additional truths to this scene. If the first actor makes the improv offer of "I'm attacking the orc with my sword!"... the second actor (in this case, the DM) could just state as their "And"... "You stab the orc in the chest and it falls down dead!" Perfectly logical and reasonable "Yes, And" response. Maybe not that interesting or compelling... but it's quick, it's easy, and the scene can now move forward with what happens with the first actor standing now over a dead orc.
However... doing it in this way doesn't make it much of a game. So rather than the second actor DM making an ad hoc "Yes And" response of saying "Orc's dead! You killed him!"... we use the dice to give us the response instead. It takes the burden of "Yes And"ing the scene offer off of the DM and puts it on the dice roll. The dice roll gives us our "And". Yes, the attack roll hits... And... the attack does 5 points of damage... And... the orc now takes their turn and moves over here... And... the orc swings it's axe at the ranger and hits... And... the swing does 10 point of damage... And... it turns out the axe was poisoned, so there's a chance the ranger might slow down their movement because of the poison... And... the ranger failed their saving throw so the ranger is now slowed... And... the ranger on his turn decides to take out a dose of antitoxin and drink it... And... And... And... and so forth.
These are all things that the two actors could just completely make up and offer up to each other back and forth in the scene if they wanted to. They could completely improvise the whole battle, giving and taking highs and lows, building off each other's actions to create really thrilling drama. But... that can be difficult. And tiring. And sometimes even less creative. Which is why pushing off some of these things onto the dice to dictate what happens can be very helpful, very original, and sometimes even more exciting (because now neither actor knows what is coming up.)
But this is also why I'm not so beholden to the dice... because they are only just an addition to the work we at the table are already doing, they aren't the work itself. The dice add color to the scene, they add results to a scene we might not have thought of ourselves had we just been doing straight improv, and they allow us to win-- and more importantly, lose-- as a surprise to us all. If it was straight improv, an actor would have to make the choice to lose by offering up the losing scenario (and the other actors agreeing to it). But that can be exceedingly hard... arbitrarily deciding in the middle of a combat scene "You know what? I'm going to fall unconscious now"... knowing you are taking yourself out of the fun of the scene. No one wants to leave the scene if it's fun! They want to keep playing! And so... we let the dice push us in that way. Let the dice make the offer of "You're unconscious now". And that makes it easier for the actors to accept and go along with it.
To make a long story short-- (Too late!)-- the game rules of D&D acts as our additional scene partner in our improvisations... making offers of how our scenes could go, usually in directions we may not have thought of ourselves. We don't NEED them to be our additional scene partner... but it can be awfully fun to have them.
When I'm playing, I have no problems whatsoever making poor or deadly choices if I think they make sense for the scene... and that comes directly out of my improv background when I've had no choice but to "play to lose" in a scene just so it can end.
