The level of overdesign required is huge. Consider how annoyed some people got at Storm King's Thunder, where large sections of the adventure would never be used in a single playthrough.You can design a completely sandbox style adventure where you drop the PCs into a setting and tell them to make their own fun, but that's hard to fit into a single adventure. Worse still, that's very hard to plan for. The PCs might decide to become bandits, they might opt to get into local politics, or they might burn the place to the ground, with absolutely no guarantee they will go the the Temple of Icky Badstuff that fills up a whole chapter of the book. They also require a lot of player buy-in and motivation to run, and that's not always a guarantee.
I'm very fond of books that present a central hub with short adventures around it (e.g. Sly Flourish's Fantastic Adventures), and that's also something the D&D Essentials Kit tries to present.
In my home campaigns, I approximate this with lots of short adventure bought either from the DMs Guild or through older products.
But when you start moving into player-driven stuff, then it's hard to present that in a good form.
Cheers,
Merric