I played 1st edition AD&D from 1979 until 1989. I dabbled in 2nd edition for a year or two and then gave up on it when I went off to college/wife/job/kids. Fifth edition and a cohort of sound players brought me back. Fifth edition is outstanding--I'm a big fan. However, one thing I don't understand is the decision to eliminate stand alone modules. Maybe this was done in 3rd or 4th edition--I'm not sure. Our group did the Tiamat arc in 12 months, and now we're doing Out of the Abyss. To be honest, I want downtime. I want to build a keep and roll on a followers table.
Technically the first "Adventure Path" was the Dragonlance saga in late 1st Edition, although it was not thought of that at the time. Prior to that there were linked modules, but these were sequels or thematic ties and not strong story. Like the Giants modules or Slaver series.
The concept of a level 1 to 20-ish campaign was an idea that came out of the marketing and design team early in 3rd Edition, and the initial adventures were meant to be that. But because of turnover and layoffs, the final product was very different and not really an Adventure Path and more a series of events that followed each other.
Adventure Paths were really first done by the Dungeon magazine team (when it was licensed out to another company, Paizo Publishing), spawning out of talk from the people who had the original concept.
Because they were a lengthier and more complex, APs had more story: characters, plots, twists and turns. So more people read them almost as serial fiction. Sales of the Dungeon issues with AP adventures sold really well, and through conversations with fans there was a lot of people who only bought the AP issues, skipping issues inbetween. By having an AP adventure each month, the publisher incentivized people to subscribe to the magazine, which greatly helped Paizo's business as they had a guaranteed amount of money each month. So APs became a big part of their business and a selling feature.
(4th Edition also did something similar with its first series, that were ostensibly connected… but not really. They also have the Scales of War Adventure Path in digital issues of Dungeon, mimicking the ones from the print magazine, but this was the first attempt by WotC and had some problems.)
When Paizo Publishing lost the Dragon and Dungeon licences they went independent and launched their own product line. And they went with Adventure Paths because they were popular, it was content they were used to producing on a monthly basis, and it was something they were good at producing. And they could use some of the ideas they had already been brainstorming. (It was also a business decision. They'd taken subscription money from subscribers for months of future issues that they now couldn't deliver, and thus had to send something in place of the magazines each month, and refunding everyone would bankrupt the company.)
Since then, Paizo's APs have been very popular. Because of the reduced prep, world expansion, and story aspect (a DM plays it twice: once in their head as they read, and again at the table as they play).
Now WotC is doing their big APs. Or "storylines" as they refer to them (as "Adventure Path" is synonymous with Paizo's products, but apparently not trademarked - ). The big storylines are advantageous for them, giving them larger stories to tie into their licenced products (miniatures, the video games, accessories, board games, comics, novels etc). And it also gives D&D more stories, which provides inspiration for potential films or television. The storylines also provide a shared narrative, so everyone can talk about the same story and events, even if they're consuming the game digitally, via public play (the Adventurer's League), or the storyline modules. You can get together with other players and talk, sharing stories and comparing events.
The storyline adventures also encourage repeated play. One of the strengths and most fun elements of D&D is the campaign, the continuing storyline that evolves over time. When you have a short module, you play that and you're done. It's good for a one-shot. But it's also easy to just end and walk away, like having finished a board game. "Cool, that was fun. See you later, maybe we can play Settlers of Catan next."
With a storyline adventure, you play and then end and there's still more story. There's a cliffhanger! It encourages people to come back again and again to finish. Then they're hooked for life. (Cue the diabolical laugh.)
There's also the game store issue. Big adventures are large books. This makes them easy to display either frontways or sideways, as they have a large enough spine to be visible.
Smaller products, softcovers that are thinner than 96-pages, tends to vanish on the shelves when turned sideways. You can't see what they are at a glance. They're harder to sell.
-edit-
And there's also price. A 256-page hardback adventure costs $50. But eight 32-page adventures might easily cost $10-$15, for a final price that's much higher.
Paizo continues to sell their Adventure Paths as individual issues, and has noted that sales dip between every issue, with the first of the series selling the best and easy successive installment selling slightly fewer copies. Which means they make less money with part 6 than part 1 despite that part taking just as much work (and possibly being the hardest to outline, design, and edit).
-end edit-
For people who really want small adventures, there are some options. First, are classic adventures. It's super easy to update adventures from 1st or 3rd Edition - and sometimes even 4th Edition - to 5th Edition. You can almost do it on the fly.
There's now also the DMs Guild, which is selling 3rd Party adventures as well as the Adventurer's League adventures. Lots of various adventures there. And you can get digital copies of classic adventures to quickly update.
