Good analysis. I prefer the old way, but it was never super-popular. On the other hand, running the same style all the way to 20 doesn't really work either.Since 3rd edition was released, high level play has been about drastically increasing HP, more powerful spells, more magic items, etc. For the first 25-ish years of the game, the mechanical power of the characters did actually stop around 9th level. Higher level characters were assumed to have more impact on the campaign world, creating strongholds, ruling over empires, even ascending to godhood.
The contemporary system has made high level play less interesting in some ways - it's just more of what you've already been doing for the entire campaign.
But I don't think games back then or games today actually use any of the high level material anyway.
What would make groups use the game material? Well, I think most campaigns end due to the time spent and complexities of high level play.
I can't imagine a game where groups would ever use it that would still fit the power fantasy theme of D&D.