When it comes to 4e, the changes were well-considered. They even wrote a whole book explaining it (Worlds and Monsters).
Explaining it doesn't mean they considered it. When your idea of "why people like tieflings" boils down to "people like playing the 'bad boy'," you're failing to consider the lore closely enough.
What actual examples of lore change/development do you think are analogous to this absurd example?
A small selection? Gnome wild mages in Krynn. Asmodean tieflings. Elemental giants. Demon-created gnolls. The Amber Temple. Gypsy halflings. The Time of Troubles. The Spellplague. The Feywild in Dark Sun. The Dawn War and Torog. Eladrin as a PC race.
Hell, this lore change isn't even
quite as dramatic as some of those - it was never explicitly stated what went into House Cannith's creation forges. Filling that empty space with dead orphans wouldn't even necessarily disrupt the themes of the setting (which can include the dangers of modernized military-industrial technology).
And why can't people just ignore them? That is, why is House Cannith turned into a villainous organisation?
That isn't a concern about communcation breakdown. (Which is what you were saying you were worried about.) It seems to be a concern about rewriting the significance of something. Which seems to assume that people are obliged to incorporate such rewritings. But why would they be?
You misaprehend. To repeat:
There is
a cost.
There's a cost involved in correcting people who come to your game presuming that House Cannith is a villainous organization because that's what the most recent lore dubbed them. There's a cost involved in correcting people who come to your game presuming that House Cannith is
not a villainous organization because that's what the original lore dubbed them. The Great Gnome Wild Mage Debate has rather clearly demonstrated that, I would hope.
Thus, with this lore change, it becomes less clear what "Let's play an Eberron game!" means, and it makes playing "an Eberron game" more difficult than it otherwise would be.
I understand what you are saying, but it literally applies to every change they make. No matter what the change is, there will be those that like it, those that don't like it, and those that don't care one way or the other. How are they supposed to make any kind of decision based on that? I mean, any change should be well considered, but how are they to take into consideration a cost that happens to literally every change?
The easiest way is to transform your process from "change" to "add." If you've got a cool new idea for tieflings or gnomes or warforged or whatever,
add it to the lore, rather than revising lore to be exclusive to your new version. I'd totally play an Eberron adventure where we infiltrated the secret forge of a rogue House Cannith artificer and uncovered an orphan-murder-artificial-solider facility. I'd totally allow a warforged subrace born of orphan-murder in my Eberron game.
When you're adding, you do still need to carefully consider your lore (maybe gnome wild mages still wouldn't pass muster!), but it becomes more flexible, since it is relieved of the duty of setting expectations for the players.
And you can still change. When you need to. When it's well-considered. When the benefits are big enough. Just maybe not whenever you think you have a Brilliant Idea.