I bought the Out of the Abyss book and am not a fan. It's too open ended and too difficult. I think it would have been better as a source book/ campaign setting on the Underdark. Honestly I'd rather have smaller modules like they made back in the day. Why are there so few books/products is what I want to know. Is it because this generation wants to download everything for free?
From what I understand, it primarily has to do with economics. The cost of developing and publishing a small module is not all that different than an adventure path. But the single large book has a higher profit margin than several small modules. Which means that you have to sell fewer copies of the large book than the small modules to turn a profit.
There's quite a bit of information around on the fall of TSR and a large amount of the problem was the glut of products they released. It wasn't just a question of quality, or even too many products, it was that the products they were releasing weren't profitable anymore.
I'll point out that while WotC is not publishing modules in the same way as the early years of the hobby, they do release about a dozen independent but somewhat interconnected short modules that relate to the main AP for Adventurer's League. These are now released simultaneously on DMsGuild for very reasonable prices. So they really are targeting that market as well, but in a different way.
That's probably partially because the markets themselves are different. Most of the folks on forums like this are the real enthusiasts that have our own particular wish list for the content and method of delivery. And like me, many of us buy the APs as well, even though I'll never run them as written.
But the AP is packaged for the more casual gamer. The groups that enjoy the game and just need an "instant campaign" that will keep them busy until the characters approach retirement level. The focus of these APs, in my opinion, is driven in part toward the sort o "video game" concept that the players should gain levels and new abilities on a fairly regular basis. To a large degree, that's a primary goal of the players. And if that's the approach your campaign takes, the APs work very well for that.
This approach is a very easy product to produce and sell. Another factor with the AP approach is that it produces an adventure that you can "win." This also plays into a large group of today's players, like folks that jump over (or back and forth) from video games, MtG, etc.
They also work well for newer DMs - they don't require a lot of DM prep, and if you follow the path from Phandelver, it makes the job of learning how to be a DM a simple one, if focused toward one particular style of DMing.
But the biggest factor is still the economics. The more products you publish, the more people pick-and-choose which ones they will purchase. Plus you tend to cannibalize your other products. In a lot of hobby industries, a new product sells something like 80%+ of its total copies in the first year. After that it's just trickle sales. But if you release a new product each month, then those products shorten the potential life of the prior products.
In addition, the larger AP is more profitable, because it costs a smaller percentage per unit to produce. So you don't have to sell as many copies to recoup your costs and start making profit. If you have a 20% margin across all products, then a product that's half the price has to sell roughly twice as many copies to make the same amount of money.
