It sounds like you're saying that moving players away from published settings is a deliberate plan of the D&D people? If I'm wrong, ignore the next part, please.
Almost. Part of what I'm saying is that the move away from
multiple published settings is intentional on the part of WotC, as is the watering down of the Realms. Neither of those are secrets. As you said, going too deep or too broad with settings has been shown to be not profitable.
The really cynical part is that I think that the way in which they're using the Realms goes beyond just capitalizing on any goodwill and name recognition the setting may have. The level of integration is abstract enough that you don't have to grok the full history of the Realms to use it, but deep enough to be difficult to decouple the setting from (definitely) the adventures and (potentially) the other source books. Thus, there's just the right amount of pressure to encourage folks to use the Realms as their setting, rather than build their own. This opens up a market not by cannibalizing the other published settings (primarily), but by going after the homebrew segment, which is a much larger segment. That might be just a side-effect, but I suspect they're smart enough for it to be intentional.
Really, it's a great business move for sustainable cash flow, especially given the sour taste a lot of folks have from the glut of crunch in the d20 heyday. I just happen to be outside the market being opened up, which means I dislike the direction for purely selfish reasons. I don't expect my vote to change the direction, but I have no problem pointing out where it leads and hoping others also dislike it. At a certain point, the game will be one that is no longer attractive, to me, and I'll move on.
The move away from published settings isn't just resticted to D&D, but rather its endemic of other TT games as well. The simple fact of the matter is that almost every game publisher has admitted that such books don't yield as much profit (if they even break even) as other kinds of books. The demand has simply been going down.
The reason, as I said before, is quite possibly tied to the explosion of computers and the internet. Research is far easier now than it ever was. Getting maps are easy. And so on. That part is easy - you sit down with your phone while waiting for an Uber or while in the dentist office, and look up wiki articles. Writing the story parts are harder, however. Story ideas are the demand in many games on top of mechanics.
I think the Indie trend is largely to move towards supporting homebrew or mini-settings (see Fate Worlds). This is probably, in part, because full settings don't sell as well and partly because the Indie segment is somewhat defined as an alternate or reaction to trends in the larger publishers, which is to couple a setting to the rules, even if that's just Golarian and the Realms.