Because sometimes new things are cool and interesting. If they were to make a new setting with fresh ideas, it would get more hype than just a reprint of a M:tG setting or redone setting from a previous edition and bring something new to the game.
I saw you discounting Wildemount, but you really can't. It's a new setting to D&D, it's not "tainted" or unusable to a large amount of gamers because it's affiliated with CR.
Oh, I did not mean to dismiss Wildemount as a setting, I am really very fond of it and have done a campaign in it already. I agree that it is a new setting in D&D, and its great that we have it officially now. I would love to see more 5e content for Exandria. However, WotC did not make it. It is a homebrew setting that has now entered the D&D "canon," which is great, but IMO it doesn't count as a "new setting."
A new setting for D&D, IMHO, would be made by WotC, possibly while collaborating with the original designer (like how Eberron came about).
So, what market niche is missing? What large group of players has not been served, and will be served better by a completely new setting than updating a old one?
I'm not sure what specifically the community would like and which setting would be best if brought to D&D, but here are some examples of niches that could have a setting:
- Modern. With technology, the internet, and magic. Like this.
- Underwater. A water world with aquatic races and peoples.
- Feudal/political. Different conflicting nations in a medieval type world. It would be different from the Forgotten Realms and Greyhawk, where magic is extremely rare and possibly even seen as a myth or legend.
- Psionic Eberron. Basically a world where instead of magic and artifice being their technology, psionics is their technology.
- Lovecraftian world. Basically an aberrant-horror setting.
There's a lot of others that I'm sure I could think of if I were to give more time to it. There is a lack of a world that has a major dream-realm connected to it (Dal Quor doesn't count, I mean a dream realm that doesn't have worm-crabs in it), or a medieval type fey-touched realm, and so on. There are design spaces that could be filled with new worlds. If you can't think of any, you need to think outside of the box.