But the core problem is that a novel is plotted. You can't have a plot AND have agency to break the plot. You can maybe have side adventures that are unrelated to the plot, or maybe you can encounter aspects of the plot in a differing order.
When I say more like a novel, that doesn't necessarily mean that there's a predetermined plot already written before you even start. Not even all novelists write that way. It can mean having the skill of pacing the session, and having the tension of a novel. It can mean more complex character interactions, like a novel, which don't have to be pre-plotted. It can even mean having the skill to improvise plot points literally as they're occurring. But it's a question of DM skill, certainly, and those who lack some of those skills have been provided tools in the form of pre-written adventures. But, that's a two-edged sword, as we've seen many of them turn into railroads when the rubber meets the road and people actually run them. But even that is a question of DMs lacking the skills to run an adventure
without already knowing what's going to happen, or being unable to improvise when things happen that they don't anticipate and can't deal with.
Sort of. A published adventure has the wrong reputation. That is, something where all the work is done for the GM. Its anything but that. If folks would view them more as a recipe as I noted above and understand they have to cook the meal and their proficiency in kitchen is going to have an impact on the overall result.
I think that's part of the problem too, which is sorta what I was talking about just above. Modules have been written for DMs who want a certain experience but lack the skill to pull it off. They've kind of got training wheels on them, for lack of a better term. But many trad style GMs are happy to keep riding with the training wheels on indefinitely, and don't want to actually figure out how to balance the bicycle without them. I understand why modules are written the way that they are, but GMs who run them just directly as written doesn't really provide me with a great experience. I find modules useful to me when 1) they're interesting to read, so if nothing else, at least I enjoy reading them for its own sake, and 2) if they have interesting characters, situations or other elements that I can use, even if I take it out of the context that it's presented in. But if it's presented without any context, it lacks element #1, and therefore sometimes isn't recognized as an interesting element in the first place.
To continue your metaphor, a module is like a recipe in the sense that it's not just a list of ingredients. It's also the instructions on how the writer thinks you should proceed with the ingredients. Sometimes you take some of the ingredients and sauté them together, then remove them from the pan, put them with other ingredients in a bowl, and then mix them, and then pour the whole thing into a baking pan and bake it, etc. A really good cook who knows the group that he's cooking for can look at that and immediately know, for example, "y'know, I don't like broccoli florets texture here, so let's replace them with chopped broccoli, and we don't like mushrooms, so let's not use those. Chicken isn't as interesting in this recipe to me as shrimp and scallops would be, and what this whole thing needs is some lemon zest and paprika, which the person writing the recipe didn't even think of, etc. etc."
I guess ultimately what I'm saying is that nothing can replace GM experience and skill. There are a lot of guard rails in place to keep unskilled GMs from going completely AWOL, but if you've got all that in place and a GM without a lot of experience and skill, you'll still only get a mediocre game. Nothing replaces real world experience and ability to make something of the material beyond what the writer was able to articulate.