I've never understood this mentality. Probably never will.
It's YOUR game. I, as a DM, have made plenty of changes to my home Realms, and I've never felt that I can't put things in, or change things, or that I'm a slave to the canon of the Realms.
It's a matter of where to draw the line between "I'm playing FR, but changing some things" and "I'm home-brewing and pulling some FR concepts". I'm not a Realms player, so I'll use Greyhawk. Even then, I only consider myself a casual fan.
If the DM tells me we're using Greyhawk, that implies certain things to me. If I say my character is of Flan decent, was born in Verbobonc, served in the Shield Lands against the forces of Iuz, and settled in Dyvers, the DM should understand that. If he says, well, IMC Iuz was defeated twenty years ago and that's where the fairies live, now, I'm okay with that. If he says the human races are more polarized and there are no Flan west of the Nyr Dyv, I can deal with that, too. Dyvers was leveled? Okay. Verbobonc was a silly name and it's called "Fredville"? Fine. All of these? Hang on, now. How much info
can I assume from the box set? Likewise, if he says, "What's a Flan?", I'm going to be more than a bit concerned that he's using just a map with city names on it.
I really don't care what color underwear Mordenkainen wears. I think the people who do are a bit... unbalanced. Somewhere in there, though, is a point of miscommunication waiting to happen. Sometimes, the expanded history for a setting is pretty cool, too. It's pretty tempting to say, "Let's just use that, unless I say otherwise," which can just snowball the problem.
I don't want to set myself up for that sort of issue. Especially, if the history and extra meat really is pretty cool. But, it's a set up for bad assumptions.