From my perspective, very long campaigns were common in the past simply because there wasn’t much choice. There were only a few game systems, essentially one per genre, and each of them only had one or maybe two supported worlds to play in. So GMs would craft one world and play in it and the cost to create a new one would be more than the value of the new stories that could be told in it.
D&D, in particular, allows for a very wide variety of play styles, and the generic fantasy worlds that, as far as I can see, 90% of very long campaigns run in are particularly adaptable. So it’s easily possible to play and not get bored with the same old thing.
in my case, I ran a Middle-Earth / Rolemaster campaign for about 5 years, and a D&D 3.0 / 3.5 / 4E campaign for ages (you can work out how long based on the fact that it started in 3.0 and finished in 4E).
The investment cost of these systems is high. We’d never have switched to HERO or GURPS to play the same sort of game — and the D&D generic fantasy worlds are pretty much all alike. I’m confident I cou,d take a character from one of my long fantasy D&D campaigns and run it in anyone else‘s fantasy D&D campaign with a few changes, but not changing the essence of the character.
The Living Greyhawk and similar campaigns are examples of that; a shared world with millions of characters who could just as easily be run in home games: D&D campaigns are generic campaigns that have wide appeal and adaptability.
Nowadays, however, we have more systems that can be learned rapidly and that are worth switching too. If I want to play a vampire hunter, I don’t have to choose between cleric and paladin — I can run Night’s Black Agents and have a far more suitable system for it without a huge set-up cost; the resources available are enormous.
Of course, for many people, they enjoy doing the same sort of style of play over and over. There’s an analogy with dining out I’ll make: A lot of people, when they go to new restaurant, try their favorite dishes. They get their pleasure from repeated enjoyment of a known good thing. Other people try the restaurant’s speciality, or something new — they get their pleasure from a new experience. Gaming is similar I think; some people enjoy the known goodness of an experience that is almost certain to be fun. Others are willing to be disappointed because a new good experience is what they are looking for, and so they want to try new systems.
Those who play long fantasy campaigns are in the “known goodness” camp. i’ve played in such games and campaigns for 30 years, and they deliver consistent enjoyment. But I also play/run shorter 2-4 year campaigns with other systems, and while they take more effort which is hard to sustain for decades, they have delivered a stronger experience for me.