D&D™ incorporates the world as laid out in all the various source-books and supplements.
<snip>
3e and esp 4e had very tight reins on what the DM could do in terms of rules and even lore
Both these claims are strange to me.
D&D (TM)
can't incorporate
the world as laid out in all the sourcebooks - it can't be true, for instance, that the events of some module are happening in both FR and in GH, but the modules contain advice on adapting to other campaign settings.
In other words, there is no single world laid out in all the sourcebooks. They're books of rules and advice for playing RPGs - and they present setting material as an aid to that. They are not catalogues of a Platonic multiverse. For instance, the statement "If a GM wants to run this adventure in Eberron, change ABC to XYZ" isn't a statement about the content of the multiverse. It's real-world advice to a prospective game referee.
The statement about "reins" is also odd. First, it seems contradcited at least in the 4e case by the inclusion, in the DMG, of a page of advice on houseruling. (I seem to recall something similar in the 3E DMG.) There are also the comments about the GM's power to author settings that I've already posted once or twice upthread.
Second, many people changed rules and lore in 4e and the world didn't end.In my own case, I strated my cam[paign using Night Dark Terror, a B/X module, and for 30 levels the campgin has not moved to any part of the mortal world beyond that campaing map. I ran G2 as a mid-epic adventure (with levelled-up frost giants) located in the Feywild. Nothing bad happened as a result.
Chris Perkins posted a regular column on the WotC website where he talked about events in his own campaign, which included a non-default hisgtory, goeography and cosmology, including a nondefaault treatment of mind flayers.
The same was true for 3E - the only (brief) 3E campaign I ran started in The Bright Desert, using an old White Dwarf mini-module (so statted for eithe rOD&D or AD&D) then shifted to Castle Amber, and if it had kept going presumably would have ended up in Averoigne.
These "reins" that you speak of are pure phantasms!
True, but 3.0 did usher in the "there's a rule for that" era of D&D which intended to create a "unified experience" via consistent rule application. Its also the beginning of the "everything's core" mantra 4e wholly embraced and it slowly shifted to "player empowerment" as time wore on.
And this creates "reins" on lore how? Prevents changes to the rules how?
The only 4e class that has significant setting detail built into it is the warlock. Much like the 5e warlock, in fact! But even there it's trivially easy to reconceive the nature of the warlock's patron if a table wants to do that.
playing Scarred Lands using the 3.0 PHB and DMG is as much "D&D" as using Pathfinder to play Eberron, but neither of them come as close to playing D&D™ as running Forgotten Realms with the v3.5 core books would.
But how is this meant to be reconciled with the default gods in 3E being from GH? Or with the default planar structure being different from the 3E-era "world tree"?
Canon should be the starting point of any discussion on lore.
Why on earth should this be so? If someone posts a thread saying, "I want to run a vampire mystery - any suggestions for ideas, published modules, etc?" why would anyone think the starting point should be canon? Maybe someone just published a cool module on DM's Guild, or under the OGL, and that should be the starting poin
The ideal would be to keep things as internally consistent as possible while improving on things that haven't worked (creating a continuity of experience across editions) but obviously, conflict will arise.
(Tangent: I think a lot of the rejection of 4e comes from the fact it intentionally and willfully broke this "continuity of experience" in terms of both rules and lore. It may have created a game that was interesting or even superior, but the intention breaking of both traditions to the amount it did is the heart of the cry of "Not D&D anymore").
My issue with this is that it implies - as was often asserted - that those who liked 4e hated D&D and hated its traditions. It's an attempt to stake out one particular approach to D&D (one that I, personally, associate with the highpoint of the 2nd ed era) as D&D per se.
Wagner set out to perfect the symphonic tradition of Beethoven. Some critics think he succeeded. Others think he didn't. But a bald assertion that "the problem with Wagner is that he rejected the tradition" would be ludicrous.
Mutatis mutandis for 4e and D&D. There are different views on what it's tradition is, and hence on what counts as developing, perfecting or (whether deliberately or inadvertantly) breaking from that. Simplistic caricatures of those who happen to read the tradition differenlty from you don't contribute anything useful to reflections on and discussions about the nature of D&D.
My point remains the same; it has to be the starting point for a discussion on the game, but you are always free to do what you want in your own game. Just don't tell me since you find it worthless, its worthless to everyone.
You don't find this a touch ironic - that you are asserting that, because you care about D&D(TM) and it's canon - as you conceive of it -
everyone should, and should use it as the starting point for all discussion?
D&D began as a game about making stuff up: worlds, characters, monsters, the events that occur to them, the places where those events happen, the tools (spells, magic items) that are used in undertaking those events. Those of us who think that lore and canon remain in the service of that creative purpose, and not ends in themselves, aren't some deviant RPGers who don't really understand the hobby and whose views don't count!