I think you’re attempting to infer FAR too much from this passage of general advice to newish players, particularly when Gygax was also advising DMs to make sure the dungeon denizens react logically the situation as in DMG pages 104-105.
As I posted not very far upthread, between the PHB and the DMG you can already see a tension between pressures of gameplay (which require an artificial dungeon environment) and presssures of verisimilitude (which push towards a "living, breathing worlld"). But modules published c 1978 were not "living, breathing worlds" in the modern sense. They didn't have NPCs whose friendships, connections, fields of action etc were remotely realistic. They have NPCs who living in holes in the ground, with no visible economic means of support, and whose response to dungeon raiders depends primarily on a reaction roll. (Consider the hobgoblins in the example of play in Moldvay Basic.)
Without this artificiality, that style of play can't work, as the players can't scout, collect information and then plan and execute raids.
I think that increase of verisimilitude is just the effect of people having played more.
<snip>
When Gygax was writing the 1e books in 1976 to 1977, he'd been DMing for 4 to 6 years. I started playing in 1992 when I was 13. I had 6 years of DMing experience in University. Now I have over two decades of experience running and playing D&D. Gygax was a creative guy, but me right now probably knows more
I think that's part of it. I also think that the sort of experience RPGers were looking for changed - the original designers and audience were wargamers, who might be expected to tolerate an artificial set-up as part of the context for gameplay.
But as the "story" part of the game looms larger among the player-base, and the PC increasingly is seen not just as a playing piece whereby the player gets to insert him-/herself into the fiction, but an imagined person comparable to a fictional protagonist, those issues of verisimilitude etc loom larger.
I mention Runequest because I think it's the first fantasy RPG to self-consciously make this contrast -
experiencing a fantasy world and story, rather than playing a wargame - the heart of the play experience. (Compare Tunnels & Trolls, which doubles down on the artificial and absurd elements of dungeoneering; or Moldvay Basic, which has a foreword about the protagonist slaying the dragon tyrant, and thereby freeing the land, using a sword gifted by a mysteriuois cleric, but the gameplay of which (as presented in the book) doesn't remotely support any such thing.)
So the change is an issue of experience, changing taste, changing membership of the RPG community, etc. The contrast has both temporal dimensions (eg 1976 vs 1986) but also system dimensions (in 1986 there were still people playing T&T as well as RQ and DL) and cutlural dimensions (consider contemorary OSRers, for intance).
I'm amused by how you keep saying "RuneQuest and modern games" when RuneQuest was first published in 1978 and predates the 1e DMG...
Well, I did say "or", not "and"!
Like "modernity" as sociologists and historians use it, I'm trying to get to something which has temporal but also cultural, structural etc dimensions.
I was actually thinking of Against the Giants where there's a random chance certain changes happen.
<snip>
Planning is impossible if things are shifting continually. But there's already a chance of uncertainty in random monster encounters, which make planning tricky. By the rules, there's meant to be a chance that plans get derailed due to bad luck. A skilled DM will find a middle ground, having most things feel similar but changing a few details to reflect the actions of the players or passage of time. Some changes due to having a "living world" might be beneficial to the players.
<snip>
The modules never account for big NPC monsters sleeping or going to the washroom or the like. They're always found in their throne-room or dining halls. There's no advantage for attacking at noon or midnight. Which is something you'll quickly consider as an experienced player, but might not as a newbie. But the module can't give two complete descriptions of the dungeon for a diurnal and nocturnal attack.
Yes, these are exactly the sorts of things I'm talking about - Although I'm not sure what you've got in mind for Against the Giants (I'm fairly familiar with it, and have just had a quick flick through, and couldn't find anything like what you describe - but I have seen it in other modules.)
But Against the Giants does have a perfect example of your point about "sleeping quarters", though with a different fiction: Room 5 of the Fire Giant Hall is Queen Frupy's Chamber, and it has the following text:
Any intruders entering the place will be commanded by Queen Frupy to kneel in her August Presence and state their business, so that she may fairly dispose of their humble requests. Any so foolish as to do so will be sorry, as Frupy will call forth her pets [a pair of giant weasels that are described as being out of sight when the PCs enter the room] and herself strike at the most powerful appearing of the intruders, She will strike at +4 due to her position, do +8 hp of damage . . . and a score of natural 20 on the die indicates she has decapitated the victim of her attack. She will then bellow for her serving maids [8 more giants] to come to her aid.
From the point of view of a "living, breathing world" this makes absolutely no sense. Given the layout of the place, anyone who arrives in Room 5 has already fought their way through the Grand Hall and probably dispatched the serving maids too. It's only when we treat each room of the dungeon as its own little vignette, with its own internal logic, that Room 5 can be seen as a puzzle/challenge posed by the GM to the players.
On your point about a "middle path", I think you're correct that that is what is intended by Gygax, but my own view is that that middle path is incredibly hard to tread - if all the defenders in a dungeon really act rationally, as (say) the inhabitants of a mediaeval castle might, then the PCs would have to be laying siege, not picking them off room-by-room - and I think the model of gameplay has largely collapsed under the weight of verisimilitude concerns.
That's a pretty absurd example
I agree it's absurd; it was actually inspired by an essays from c 1981 by Roger Musson, in which he posits 15 ogres holding a union meeting in the corridor as a technique for a GM to discourage the players from heading into an as-yet unmapped/unstocked section of the dungeon. The example (no sillier than some others given by Musson) illustrates the balance between absurdity and verisimilitude that seems to have been accepted (widely, obviousy not universally) at the time, but I think would be widely rejeted now - the ogres are holding a union meeting (but aren't workers) in a corridor of the dungeon (not in their guidl hall) and don't pursue the adventurers who stumble into them but then turn around (otherwise they don't do the job the GM has put them there for). These are ogres that make no sense except as a device in a megadungeon!
I don't see the game system at play here at all. You can play the DM vs Player dungeon crawl skill game with most modern systems. Conversely, many DMs playing OD&D and 1e D&D created expansive setting.
It depends what you mean by "system", I guess.
If the "system" includes reaction rolls, or relationship rules, or wandering monster/random encounter checks; these all matter to the stuff we're talking about.
Then there are advancement rules: contrast XP-for-gp, which establishes a clear measure of player skill, from the rule (very popular among Melbourne D&D players c 1990) that the PCs level up "when the GM feels its appropriate". What does levelling mean in that latter sort of game? It's not a measure of skill at all, but a GM-controlled pacing device. In a "level up when the GM feels like it" game, all of Gygax's advice in his PHB becomes pointless, even if the combat mechanics are the same as they always were.
There are some resolution systems (eg Cortex+ Heroic) which make a skilled-play game of the Gygaxian sort impossible. I ran a "dungeon crawl" fairly recently using a Cortex+ Fantasy Hack, but it wasn't a skilled play game at all. Just to give one example: the PCs had been teleported deep into the dungeon by a Crypt Thing (mechanically, when the PCs confronted the Crypt Thing the Doom Pool had grown to 2d12 and so I spent it to end the scene), and all were subject to a Lost in the Dungeon complication. As they wandered the dungeon looking for a way out, I described them coming into a large room with weird runes/carvings on the wall. One of the players (as his PC) guessed that these carvings might show a way out of the dungeon, and made a check to reduce/eliminate the complication. The check succeeded, and this established that his guess was correct. (Had it failed, some further complication might have been inflicted, or maybe the carvings were really a Symbol of Hopelessness, and the complicaion could have been stepped up to a level that renders the PC incapacitated.)
There's undoubtedly a type of skill being demonstrated by that player - a quick imagination working with established fantasy/dungeon tropes - but it's not the sort of wargaming-type skill that Gygax had in mind when he referred to the "skilled player"!
So I think these changes in game play involves a myriad of factors: expectations and preferences; system elements (action resolution; PC build and advancement; content introduction rules like random encountes and reactions; etc); non-mechanical techniques (how is failure adjudicated? how is fictional positioning established? etc); and probably other stuff I'm not thinking of at the moment.
The GM can create the setting wholly before play. They can create chunks of the setting before each session, expanding as needed. They can create the broad strokes and flesh it out during play. They can entirely improvise during play. They can tap the players to add locations and places. Or the whole setting can be collaboratively created at the table by the players.
This is all true. The question the OP was asking is - if the game isn't a classic skilled-play dungeon crawl, then what is worldbuilding of the GM preauthorship variety for?