Improvisation has been a part of the game since the very beginning. An irrefutable example is the notion of a random encounter, which in the case of 1e AD&D would have included a percentage chance of being in the lair, which means a DM was expected to be able to improvise a reasonable map of some sort (a ruined castle inhabited by a hitherto unknown orc tribe, a dank cave inhabited by some dragon or fell beast, a sealed tomb containing some ancient evil, etc.) on the spot. Indeed, a case could be made that their is an implied fully improvised campaign that can be run entirely out of the 1e AD&D monster manuals. Indeed, there is yet another improvised campaign implied by the random dungeon generator in the 1e AD&D DMG.
Randomized generation of a map is not improvisation. It is generation. It is repeating the pattern that is the game so players can game it. You should remember this terminology as you were around early on. All those DMs saying, "I'm not making it up!" DMs are never to make choices after the code of the game is selected prior to play. D&D is after all a (wildly enormous) variant of Mastermind.
So what you are actually offended by isn't 'improvisation' per se. Fundamentally, stuff that is improvised in play by some means is no different that stuff that is improvised before the session by some means. In both cases, the DM has the full power to specify what you call the invisible board.
On this point you are completely wrong. You're selling
games down the river.
Like every single game, D&D enables players to play a game by presenting them with a pattern design to decipher. Just like Tic-Tac-Toe, just like Chess, just like every wargame.
Heck, even every sport. Running isn't a sport. A race with a pre-defined track is, even if played solo.
And during a game, PC's will always attempt things or ask questions about the environment that aren't fully specified by the notes regarding the map. That's every bit as much 'going off the map' as actually trying to step into a part of the map not yet drawn.
When the players go off the map the DM must generate more, either on the fly during a session which can slow it down, or by stopping the session.
Arbitrarily making something up, that isn't part of the pattern of the game, means you are expressing babble. The only game left to the player is deciphering the language code you're using to communicate. Which might improve reading ability, but little else.
What you are offended by is the DM improvising in an antagonistic manner, either motivated by his desire to 'win' and keep the players from defeating the scenario, or motivated by some other desire to achieve a particular outcome.
This isn't about being offended. I'm not. Any DM not acting like a referee, but attempting to affect the game either favorably for the players or detrimentally is breaking their oath to be impartial. They are trying to be a player in a code breaking game they created. As the DM is only ever allowed to relate what is on the game board, a manifestation of the game, they are never allowed to improvise. They are only and ever a referee.
You associate this with 'improvisation' and 'story-telling', and therefore declare those things categorically bad. But you are confused. Improvisation can be done in a neutral, unbiased, manner as part of just "running the game".
I think your meaning of improvisation requires some explanation. Unless you're referring to unbiased, un-improvised refereeing, I don't see how what you say could be true.
And likewise 'story-telling' has been a part of the game since the beginning.
This is categorically false. Storytelling has never been part of games. Not until White Wolf published the "Storyteller system" to very disagreeing game public did anyone confuse games with stories. (Heck, storytelling as a culture didn't even exist all that long ago). To be clear, storytelling is not gaming. Code breaking is gaming. They are not even the same culture.
This was the "you are supposed to fight the bandits, not surrender your money or flee" example. The GM clearly expected the players to fight the bandits. Having the bandits take the wizard's money and leave, or shake their fists and curse the fleeing PC's cowardice, would be improvising. Having the bandits attack the wizard anyway, and suddenly have the means to catch the fellow on horseback, is not improvising - it is failing to improvise by forcing the battle to happen as planned.
That's my objection. The DM never has any expectations of the players. Let the dice fall where they may. A DM isn't playing the game, so they cannot ever "force" any action.