When WotC purchased TSR, and D&D, they did something TSR never did . . . they took the time to analyze the strengths and weaknesses of how TSR managed the D&D brand (and their other games too).
According to Ryan Dancey, the guy in charge of this analysis, ONE (not the only) of the factors was too many competing product lines, most of them build around different settings. The problem wasn't that D&D had too many official settings, but that many of those settings had extensive product lines supporting them.
Dark Sun wasn't one or two books, it grew to over 20 products. Which all competed with equally large (or larger) lines for the Realms, Dragonlance, and other lines. And each subsequent release was subject to significant diminishing returns. The Dark Sun boxed set required that you had the PHB (and I think the Psionics Handbook also). Each subsequent release required BOTH the PHB and DS boxed set . . . and the deeper you got into the product line, the more you felt you had to have the whole thing. Why purchase Dark Sun module #10 if you didn't have #1-9?
And, I'm pretty sure the folks on the D&D team today realize a few things . . .
- The TTRPG industry, going on 50 years, is still in its relative youth.
- WotC's analysis of TSR was sound . . . but not necessarily 100% correct.
- The industry back in the 90s was very different from the industry in the early 2000s, and the industry today. Times change, circumstances change.
So . . . WotC experiments with changing up how they release D&D, but slowly and intentionally (as possible). And they are also subject to the whims of the C-suite, who are prone to the kind of short-term corporate thinking that (in an extreme form) killed TSR.
WotC published settings for 3rd Edition, 4th Edition, and already 5th Edition. But they've never gone back to the model TSR used with entire product lines centered around these settings. They have not spent decades telling the fan community that too many settings kill games, rather they've given us reboots of classic settings and entirely new ones.
Now, WotC is getting ready to start a new "phase" of D&D, a new way to manage the game. They are going to lean into world-building. What exactly that means, beyond the FR books we already know about, remains to be seen. How's that going to go? Nobody really knows, we'll have to find out!
But as usual, the sky isn't falling Chicken Little! WotC isn't being deceitful, hypocritical, contradictory, or purposefully ignoring the wisdom of the 2000s era D&D team. They are adapting to the ever-changing industry in our ever-changing society to hopefully keep D&D going strong and create some really cool stuff.
Some of y'all need to relax!