delericho
Legend
They did learn from what had gone before but, in my opinion, they also introduced entirely new mistakes or missteps.
Indeed.
For instance, I always associate the term 'railroad' in the context of RPG adventures with Dragonlance. Maybe that's unfair but DL and I parted company after the first module. And when Ravenloft appeared it was as as uncomfortably incongruous to me and some of my peers as Eberron was to some players when it appeared.
The reason I generally bracket these two together is that they seem to mark a distinct shift in adventure design - from Ravenloft onwards we see more and more storytelling aspects appearing in adventures.
Now, storytelling in itself is no bad thing. And, indeed, I agree that Ravenloft is an excellent adventure. But what Ravenloft does is it pairs the lovingly-crafted and very detailed Big Bad with a very large and open area for PCs to more around in, and also the gimmick of the "fortune telling" (really a mechanism to change Strahd's motivation). Together, these work to ensure that this adventure will play out very differently for different groups, and indeed for the same group if they played it twice.
The big problems with this are (a) it is extremely hard to do well, and (b) it requires a lot of redundancy be built-in to the adventure (a typical run-through of Ravenloft will miss an awful lot of the rooms - quite different from the expectation of more modern adventures, where you'll hit almost everything).
Done poorly, or done without that redundancy, that storytelling runs the risk of degenerating into railroading. Which doesn't mean players won't have fun (a good railroad is probably more enjoyable to play than a bad sandbox), but it does negate one of the big strengths of RPGs (that players get to write their own story). And it is a structural weakness in the adventure, albeit one that will only become apparent if the PCs try to take a different turn - on a railroad that leads to a crash; in a car it's just a different path.
(Or, if you will, adding the redundancy of the "what if..." paths is like fitting airbags to a car. Hopefully, most customers will never have cause to know if they actually work or not, but the few who do will be very glad if you've made sure. Though, obviously, it's a whole other scale of things.

The thing is, an awful lot of modern adventures, including 'good' modern adventures, fall into the trap of telling their story and not easily supporting variations. Consider "Red Hand of Doom", widely considered one of the very best from Wizards - it consists of five parts that basically have to be played in order, and most of those parts consists of a "5-room dungeon" that, by its nature, doesn't allow for much variation in approach. It's actually far more of a railroad than most of "Hoard of the Dragon Queen"! (Not that I'm claiming that the latter is a good adventure!)
I wonder if some of the popularity of the adventure path format is down to an attempt, if not a necessarily self-conscious one, by designers, to ask or hope that people will accept that a certain quantity of good quality adventure design is equivalent to a historically smaller quantity of what thirty years ago was considered fresh and novel in a way that is now impossible to recapture.
I'm not sure that's it. I think it may very well be simpler than that: time-stretched DMs looking to buy a pre-made campaign off the shelf that they can run without extensive modification.
And it's worth noting that of course the AP concept isn't exactly new, given the age of Dragonlance! (Arguments can also be made for the "Slavers" series and others, but I'd rather not get into those - I'm arguing that DL was a path, not that it was the first or was unique!)
(Of course, there have always been DMs who built their own 'path' of published adventures, perhaps running A1-4, T1-4, GDQ1-7 as their campaign, or similar. That's not much different from simply plucking a formal Adventure Path from the shelf, except that in the latter the connections are already present in the material to be used.)
Oh, and cynically: one simple reason for the popularity of APs is that it's much easier for Paizo to sell a subscription to a product made of six-issue paths than a product of unconnected monthly adventures. That way, people will pick up part one, set up a subscription to get the other five parts (for the convenience), and then hopefully stick with it long-term out of inertia.