Some folks think there's a direct causal relationship between this focus on settings and the demise of TSR; I don't think it is so clear, that TSR fell apart more through mis-handling as a business, not in terms of creative output. But that's another story. But regardless, it is unlikely that we'll see that sort of thing again.
Just thought I'd get this over with before really discussing stuff...
Settings are definitely held up as one of the reasons TSR failed. They're not
the reason but part of one of the bigger reasons, which was the division of the audience. Although business mismanagement was also bad. It'd be nice to believe otherwise, but the only people who have studied the numbers and bookkeeping say otherwise, and have to be taken at their word. To do otherwise is just wishful thinking.
A really, really successful company can survive dividing its audience and focusing on multiple competing brands, but I don't think D&D is one of those brands. They can't really afford to release a book that only a minority of their audience will be interested in, and really need to focus on the majority (or largest minority).
One thing I find interesting here is that both posters are, I think, speaking truths that seem opposed but actually aren't. I will agree with @
Icon_Charlie that WotC has a tendency to rehash old material and that we haven't really seen a large burst of new ideas, at least in terms of "fluff," probably since Eberron - and even that was nothing like the "Glory Days" of the late 80s to mid-90s when you had an almost gaudy display of riches in the form of new settings flowing forth from TSR: Forgotten Realms (1987*), Spelljammer (1989), Dark Sun (1991), Planescape (1994), Birthright (1995) and probably others in-between.
Risk is something you do after you're established. You don't start with risky products, you build to them after you have established your audience, found out how sales are doing, and have built some trust. Paizo didn't start with
Iron Gods but slowly did more and more experimental adventures until they felt it was safe to do a sci-fi/fantasy mash-up, and even then they followed with something super classic.
At this point, D&D is really playing to its strengths and playing safe before the team can try something different.
However, at this point an entirely new world is super unlikely. WotC has a crazy amount of worlds that have barely been used. Creating something new comes at the expense of established worlds with their own fans and established audiences. It's picking a potential audience - a theoretical audience- in favour of real people. Which is a ballsy move that can backfire. It would feel a little like saying "we think this new world is better than all the other worlds".
Lots of people want "something new" but not everyone who wants that vague "newness" will be satisfied by the new world. I'm sure lots of people who wanted a new campaign setting back in 2002 were turned off by Eberron.
Plus there are trademark issues. The copyrights are fine, but undefended trademarks can expire. If they don't use terms like "Birthright" or "Council of Wyrms" or "Ghostwalk" or "Gamma World" in reference to tabletop RPGs they risk allowing other companies to claim those terms.
As you say, new ideas are hit-and-miss. If WotC is throwing its support behind a world, they have to go all in: adventures, storylines, minis, and now potential board game and MMO support. (Like when D&D Online had quests that moved from Eberron to the Forgotten Realms.)
New ideas will almost certainly be limited to storylines. Because that's safer. They have the big somewhat risky adventure arc and if it goes well great, but if it does poorly it only hurt them for six months.
An OGL would help though. Because it would allow 3rd Party Products to take the risks, to do the new things in place of WotC. Look at all the campaign settings that were spawned out of 3e. Some still have their fans (Scarred Lands, Ptolus; I'm a fan of FFG's Midnight setting), but even the ones that were only marginally successful or largely forgotten are still remembered by their fans. And all without financial risk to WotC.