That's great for us, but not so good for the companies involved: now they either need to have the best-selling items in the line carry the duds (pushing the price up higher), or they have a situation where two duds in a row cause the line to fail. (Bear in mind that they can't know when a 'dud' is coming. If they could, they wouldn't bother publishing them at all - it's not like their tried to make "The Forest Oracle" suck!)
I've snipped lots of stuff. I agree with you about the utility of those smaller volumes. I even agree that these big hardbacks are a poor way to deliver the adventure material. Unfortunately, though, I remain convinced that we're stuck with them. Again, WotC must be aware that this format isn't ideal from an ease-of-use perspective, so if it was practical to change then I'm sure they would.
I'm afraid it doesn't work that way, for the same reason that my example $50 book was "Out of the Abyss" at 256 pages and not the "Monster Manual" at 350+. They produced many, many more copies of the Starter Set knowing that it would sell huge numbers. That vastly reduced the cost to produce each copy, and meant they could sell it at a knock-down price. Heck, I wouldn't be at all surprised if that one was a genuine loss-leader for the line.
A new standalone adventure, or even an "Expert Set" to expand on the "Starter Set" simply wouldn't sell those numbers, meaning a much smaller print run, and therefore a necessarily higher price.
Sometimes, economies of scale are a great thing. Sometimes, they work against you.