I would say it isn't the direct successor, 3e was. But 5e incorporates a LOT from all the previous incarnations, including what NOT to do (I'm looking at you 4e!)...
I started D&D with the old Red Box (in Dutch) and quickly moved over to AD&D2e when it came out, we played it for a LONG time and it has a nostalgic place in my memories. But we saw many of the advantages of 3e and started playing it even before it was released (due to Eric Noah's rumour compilations), moved nicely into 3.5e and skipped 4e (I tried but couldn't get myself into it), played a bit of 5e when it came out. Took a break and started again a couple of years ago and moved into 5e 2024.
That said, I recently dove into Dark Sun to run a one-shot under 5e 2024 at some future date. For me Dark Sun is the original DS Boxed set for 2e and the products that came with it, the revised boxed set a lot less so. So when I dove heavily into the old AD&D 2e DS rules booklet from the boxed set, things that I had long 'forgotten' surfaced about 2e. Things like only certain races could play certain classes, minimum stats for classes, limited dual/multiclass combinations based on race, bonus XP for high stats, different XP tracks for different classes, etc. The less cool things about old AD&D 2e that are no longer relevant today and haven't been for over a quarter of a century in D&D...
I wonder if that's me looking at AD&D2e through rose colored glasses (nostalgia) at something I liked at the time. Or are these unimportant game mechanics I have forgotten due to disuse after a quarter of a century? For me 5e isn't perfect either (no D&D version really is), but I suspect 5e 2024 is closer to perfection then the previous incarnation, at least for me and my group...
My experience is very close to yours, except we moved to 1e first, then added stuff from 2e. Switched to 3e, and Skipped 4e entirely, partly due to adulting- and the fact that we lived 1000 miles apart at the time, lol, and partly due to it straying TOO far from the core of the game.
We're just getting into 5e. Late to the party- that's us!!
I agree 100% with you on campaign setting lore. We never did Dark Sun, but The Realms, Dragonlance, Ravenloft, Greyhawk all got "the treatment" as well. We like the version we played then, darnit! I personally LOVED the 3e Forgotten Realms Campaign Setting book, and much of the supplimental books that came after. Shortly after the Time of Troubles, or maybe even before, is where I like to play. Ditto for the 2e Ravenloft boxed set. ( mine is long gone, but that's what I think of when Ravenloft is brought up.) We played a fair bit in Mystara, as well- back when it was just The Known World. My B/X sets saw a LOT of use.
The rose colored glasses thing is pretty true, I'd say. It's not just the products that have changed- it's the way people play, and what "fantasy" means to them that have changed dramatically as well. The mechanics were just what we had at the time, and what we were used to- and rule 0 was extensively used. I dare say, that while we used whatever AD&D books we had on hand, I cannot say we EVER played it strait. Our game was a kit bash- we just added what we liked from newer books as we got them to our game.
That said, 3e was remarkably like "our game" just powered up a fair bit, and with a lot of now why didn't WE think of that??!!. 5e distilled all of that, streamlined it- and added more new cool stuff- so it is right up our alley!
I do find myself adding in more and more stuff from earlier editions- mainly monsters and magic that didn't make the cut. As for lore, we never adopted the contemporary version at all. We're just teaching the kids and grandkids to play the way us old farts did.
Sure is nice, though, to play a more or less official version- with books everyone can get and look at, lol. Instead of hauling around a huge stack of books, teaching new players 1001 house rules, and having to pull stuff out of our butts to cover occasions when the mish mash of editions didn't match up!