Because then a good portion of later releases will contain rules which are only used by part of the playerbase
And how is that any different than, say,
Arcane Power containing options for a class not featured in the PHB?
This is the entire intent of the edition. To release content not everyone will use, but hopefully to have some content in every book someone will want. So DMs can say "I'm not allowing druids in my game because they don't fit the world, but I'll use this other option in the book."
the one who use optional rule X or Y, and the value all other players get from a book goes down.
This is going to happen regardless. That's the Con of releasing a modular game: much of the content and options will not be used. No one will use everything. But hopefully everyone will use something.
Also, having to account for every optional module and combination gets very complicated fast and takes up space. Just imagine a adventure where a challenge has to list several sets of rules just to account for different optional modules (Check to open the door basic, skills, skill point system, etc...).
Adventures and the like can default to Basic and then DMs just know to substitute advanced rules. The exception would be when the adventure is built around a particular rules module, and then this can be identified on the product. Such as a big tactical adventure emphasising it uses the "tactical rules module".
This is no different than PCs having class features that aren't accounted for in adventures because the option was released after the adventure. There's always (ALWAYS) going to be some DM adjudication required.
Having a basic and advanced set of rules would split the playerbase and increase cost for WotC without much gain.
If Basic is always compatible and the majority of releases are Basic friendly then it's not a problem. Which is the benefit of having a very simple Core.
If the Core is the "Standard" version of the game then you have to release a separate incompatible streamlined version of the game; you're basically ghettoizing the Basic players by limiting the products they can use and only releasing incompatible products. They're stuck with a narrow subset of the game and you
have to release products incompatible with the rest of the game to support them. It makes the problem worse. This makes "Basic" into a limited run product line that's released and then ends, so the players who enjoy that stop spending money.
Major releases can (and should) always build upon the Basic core. So someone can buy a later accessory and add it to their Basic game, or Standard game, or Advanced game. Smaller niche products (adventures, softcover releases) can focus on one of the three levels of gameplay because they're designed to sell fewer copies anyway.
