Randomized generation of a map is not improvisation. It is generation.
To a large extent I'm happy to agree with this - even though it is technically wrong - simply because it does no harm to my point. Anything you create without prior preparation is improvisation, regardless of the technique you employ. (Indeed, the stuff you create outside of a session is still improvisation, differing only in that it is not done under time pressure.) But for now I can let this claim stand, because none of the systems I described can depend on pure randomization. You still have to produce the orc's castle or the dragon's lair on demand. You don't have a random generator for everything. And if you'd actually used the random dungeon generator in the appendices you'd know that it doesn't work without some degree of DM guidance and judgment. The thing can generate nonsense results, spatially impossible results, and requires judgment calls on how to treat all sorts of possible results (chasms, rivers, etc.). The DMG actually tells users to apply their judgment.
It is repeating the pattern that is the game so players can game it. You should remember this terminology as you were around early on. All those DMs saying, "I'm not making it up!" DMs are never to make choices after the code of the game is selected prior to play. D&D is after all a (wildly enormous) variant of Mastermind.
This is just wrong on so many many different levels I don't know where to be. First of all, there is no 'pattern'. The dungeon board is arbitrary and the result of whim. You can't define a dungeon as meeting any sort of pattern. It can be random or nonrandom. It's not confined by anything but the DM's judgment. When you compare it to Mastermind, the real difference is easy to see. Sure, the positions of the colors on a particular puzzle are random and arbitrary and the result of whim as well. But each is equivalent. No particular pattern is special. D&D by constrast inherently produces 'boards' that are special and different from all others. They are not equivalent. Each board is inherently unfair or at the least nothing constrains a board to be fair except again, DM judgment. D&D is closer in this regard to Calvinball than it is to Mastermind. The DM not only makes up the board, but the pieces of the board, the rules that apply to those pieces, and the laws that govern all interactions on it. This is particularly true of 1e, where there were few or no metarules regarding anything, resulting in adventures were every single room had unknowable rules unique to the room governing how the room worked. A good example of this would C1: Hidden Shrine of Toamochan.
And yet, for all the copious notes, there is simply not enough notes in the module to remotely decide how to resolve player propositions about interacting with the 'board' without resorting to improvisation. Nor will answers be found in the 1e DMG to questions of how much it reduces the chance of drowning to try to throw a rope to a drowning man. It's not there. It's a requirement of being a DM to be able to make stuff up. It's unavoidable, because the game isn't definable within a narrow set of rules and no set of rules could ever be complete enough to describe the game. How the heck do you think this game is decipherable when the rules set is infinite?
Like every single game, D&D enables players to play a game by presenting them with a pattern design to decipher. Just like Tic-Tac-Toe, just like Chess, just like every wargame.
Not every single game involves a pattern to decipher. Game strategy can't be equated to a mechanical deciphering of code, because few games are so mechanistic. I notice you have a remarkable fondness for choosing games that lack any random factor. Tic-Tac-Toe, Chess, and Mastermind are examples of mechanical games, which lose their savor precisely when they become analyzable. D&D is not a mechanical game. The randomness of the game defies any attempt to deduce a pattern from it. It's not merely a case of being a wargame with a random element to resolution, like say Bloodbowl, where we can apply statistical logic to every element of the game. D&D is far too freeform for that. D&D is not a strategy game. It is a game that may occasionally have a strategy minigame, but its never confined only its strategy minigames. The DM is free to improvise into the game any sort of minigame of his choosing, according to whatever rules come into his head. Indeed, improvisation is precisely what separates D&D from a tactical wargame. Improvisation in the middle of a wargame is the creative act that birthed the RPG. The exciting notion that the game could go beyond mere 'code breaking' as you name it, and become a story is what caused the board wargamers to give up the dreary plains of Poland and become excited dare we say fanatically role-players from almost the moment that this shocking revelation occurred to them.
When the players go off the map the DM must generate more, either on the fly during a session which can slow it down, or by stopping the session.
Either way would be improvisation. The DM must make stuff up through some process. And the DM is not limited in what he makes up. If he wants to have a room filled with living candy plants and a dragon that breathes soap bubbles or a room that is a pastiche of the Wizard of Oz, or an inescapable death trap, it may not be good judgment by the DM but he's within his rights.
Arbitrarily making something up, that isn't part of the pattern of the game, means you are expressing babble.
How in the world do you think dungeon maps get created if not arbitrarily making something up?
They are only and ever a referee.
Gygax himself disagrees with that claim in the 1e DMG. The DM is most certainly not only and ever a referee, but per the rules is much more than that.
I think your meaning of improvisation requires some explanation. Unless you're referring to unbiased, un-improvised refereeing, I don't see how what you say could be true.
My meaning of improvisation requires only a dictionary. Your meaning of improvisation is increasingly reliant on nonsense and contradiction. Explain how the map can be extended without application of DM judgment and choice? I can certainly see how application of the rules can be done without bias in most cases, assuming of course there are rules covering the situation, which isn't normally the case in 1e or OD&D, but there is no way that its possible to create the game world without bias.
This is categorically false. Storytelling has never been part of games. Not until White Wolf published the "Storyteller system" to very disagreeing game public did anyone confuse games with stories. (Heck, storytelling as a culture didn't even exist all that long ago).
For someone who is quick to claim something is false, you sure have a way of stringing together a bunch of obviously false statements that betray just how limited your gaming experience is. Even were we to except your utterly ludicrous claim that storytelling isn't part of D&D (ever played I6, UK1, DL1, I3-I5, etc.), and even if we were to accept your ludicrous claim that storytelling wasn't what drove D&D from the beginning (do you know anything at all about the Blackmoor campaign?), White Wolf was rather late to the story focused gaming table, already occupied by games like C&S, Pendragon, and even to a large extent Top Secret and Marvel Super Heroes for crying out loud. There are plenty of academic sources that make it very clear that from the inception of the idea of the RPG, the idea that the game was telling a story and could be used as a collaborative story telling medium has been a part of the game. And your idea that the DM is creating nothing with no more story meaning than the pins in a game of Mastermind is ludicrous. D&D content isn't a couple random colored pins - but a setting, with characters, with motives, ands with events.
And storytelling as a culture has been around since at least the time man invented fire.
To be clear, storytelling is not gaming. Code breaking is gaming. They are not even the same culture.
D&D has no code. No rules for constructing this coded board you think exists actually exist, and to the extent that you could construct such rules they would create a game obviously and inherently inferior to D&D precisely because the attraction of having a DM is that the DM has the power to create non-mechanistically and escape any limitations of a code. People don't play D&D because its code breaking. They play it because it isn't.
A DM isn't playing the game, so they cannot ever "force" any action.
Sure they can. Any DM worth his pizza can between sessions construct a board such that actions are forced. The question is not whether he can do so, but whether he ought to do so. And there is no science to that. No rules can tell the DM whether or not his board is too linear or too broad. He has to use his judgment.