You may hate to hear this, but setting books aren't meant to be read like novels. They're meant to provide background information for players and GMs. You may like reading them as novels, but you're "using them wrong," so to speak.
And since many people no longer seem to care about or even want meta-plots in their game settings, there's no reason to for companies to continue being beholden to them--which means freedom for the writers (and players) to go off in other, potentially more interesting directions.
Ravenloft. You may think it was just an addition when it went from one adventure location to an entire setting, but that was actually a huge change--it altered everything about the setting, including Barovia. When it went from a bunch of domains to a more unified whole, with trade and relations between countries--back in the middle of 2e, with Domains of Dread--that was a huge change, because it altered how the setting was actually meant to be played. When the Van Richten Guides came out, this was another huge change, because again, it altered how you were meant to play the game. You may not think so, because you read the game books like novels, but the actual games? Huge change. It "invalidated" earlier methods of play, because it was no longer a "Weekend in Hell" setting, no longer a setting where the monsters were just monsters.
You bring up the Blood War. Yes, that--and changes made in other editions--does change demons considerably from 1e. They went from "free range" monsters who may or may not have been browbeaten into service by a demon lore to troops in a war, which alters what passes for demon society considerably. In 1e, they were specifically created out of the souls of chaotic evil dead (as per the entry on manes). In 4e, demons are (IIRC) corrupted elementals and/or born or created directly out of evil. That's a huge difference in them.
Planescape "invalidated" earlier gameplay by putting the emphasis on Law and Chaos instead of on Good and Evil.
Even minor bits of lore--like 5e deciding harpies were the descendants of cursed elves--"invalidates" previous lore by deciding that this is how harpies are now.