All lore is tied to a setting, even if it's just "the world of D&D" (i.e., the lowest common denominator of Greyhawk, the Realms, etc.). What if Orcs in my world are a proud but not always chaotic evil race of seafaring warrior-poets? What if I want Dwarves and Drow to be combined into one race, and have them be slaves of the evil High Elves? Both of these are actual things from a D&D setting I'm working on, and I don't feel that the classic D&D lore of orcs and elves is oppressing me. I can just ignore the lore if I want to. It's my game, after all.
Yup, totally agree. Now, if you told me that core D&D had to include your version of these races, I'd tell you the exact same thing. Please keep your peanut butter out of my chocolate.
The simple fact is, you can either have a lot of detail or a little detail.
If there's a lot of detail, DMs can be free to use the detail that's there, or ignore it if they think of something better.
If there's only a little detail, then every DM has to come up with details for every monster they want to use. See the problem there?
Except there is an additional problem that you're ignoring. If there is a lot of detail then every single subsequent presentation of that monster must adhere to what came before it. It's inherently limiting. All yugoloth must hate gods, despite the fact that that never actually appears outside of a specific setting. To the point where setting fans complain if it's not included in core.
Not everything is core. It's not like there's one "D&D minotaur" that is the only kind of minotaur in all of D&D. The core books can describe a default minotaur, and Dragonlance can say "minotaurs are different in this world." And that's fine.
So, why can't Planescape be the same? D&D has a default Yugoloth, or Slaad, or whatever, and Planescape has a different one. We could have Aberration Slaad in core and Outsider Slaad in Planescape. What's the problem?
You know how much effort it takes to strip out setting specific lore? None. Why does it hurt you to have that detail in there? You can just ignore it. You don't even have to read it. If you already know what a daemon is in your setting, you don't need that lore anyway.
It hurts me because every single setting book published must adhere to that lore. Every Slaad must be a Planescape slaad. Every Yugoloth, demon or various other outer planar creature must be a Planescape compatible creature. Why? Why can't core demons have nothing to do with the Blood War? Why do Succubus have to be demons? How does it bother Planescape fans if Eladrin in core are blink elves, but in Planescape they are elf angels?
Why do Planescape fans get to dictate core elements when no other setting fans do? After all, as you say, you can just ignore it. It shouldn't bother a Planescape fan in the slightest if a Demon and a Devil and a Yugoloth are used in the same encounter in a module, on the same side. They can just ignore it if they want.
But, we we ram Planescape elements into core, we cannot have that encounter in a module because it would violate the Blood War elements.
I'm still utterly baffled why fans of a specific setting get to determine how core looks.