I believe there are two, or maybe more, schools of thought about adventures. When I was running 1e most of the modules had a very brief introduction that gave the DM an idea of what was happening behind the scenes. From there the DM pretty much extrapolated the details himself, making them up as they went. Any real meat to the "story" was created by the DM either ahead of time or on the spot. Room descriptions were sparse with just basic details and contents. Some rooms would have more details such as tactics of reinforcement, etc. Monster statblocks many times were simply a name, quantity and number of HP. Take a look at the amount of "content" packed into a 32 page adventure of that time and you see how little detail you really got.
Then came Ravenloft, and Dragonlance, the good and the bad. In these there were much more details to the "story". In Ravenloft the details of Strahd's downfall are interwoven almost seamlessly into the discovery/exploration within the module. His diary has all these details but the players have to find it to get those, but since the DM has those details ahead of time he can play Strahd in the tortured soul mode that is befitting the character. However, the room descriptions and monster stats stayed pretty small. Look at the density of the castle for an example. The module is still slim (I think it was 32 pages) but the castle has hundreds of little nooks and crannies to explore. The map, IMO, was a work of art and really cemented my love for isometric views.
Then there's Dragonlance. In this case the story is "interesting" but it's all the story of the NPCs... Boring... Of course this is due to the fact that the story is actually a telling of the War of the Lance narrative with the PCs encountering the Heroes of the Lance as they are making history. Wait, aren't the PCs the ones that should be making history? Bleh, in any case the story is very linear and railroady. Important NPCs can't die, they will/might respawn again in another module of the series, etc. Still the room descriptions are very short and narrow for the most part. Look at the Tomb of Kharras (sp?), or Pax Tharkas (sp?) as an example of room density. Overall Dargonlance is my least favorite module series, but it is the one I mined the most for locations, NPCs and maps. Xak Tsaroth (sp?) the sunken city is a formidable adventuring location.
I skipped most of 2e, but I do have some adventures from that era. It seems to me that the "background story" started to take a front and center position instead of the "story of the PCs". I can't say I've seen anything memorable about any 2e adventures. It was during these modules that the room descriptions seemed to start getting larger with lots of fiddly details, but I can't really lay all blame here. The worst offender is coming up.
Then in 3x the adventures became massive mostly because of the monster stats. This consequently lead to the delve format, which is the real culprit of adventure size and detail. Each significant room encounter is given this 2 page spread that has important information about the combat. First "problem" is that most assume combat in every instance. Granted this information is useful because monsters were so much more complicated in 3.x. If the DM was not aware of "useful" tactics or use of spells looking stuff up was a hindrance, so the format is definitely useful. The format also was helpful with presentation of miniatures as large maps of encounter areas were usually included in the adventures. But no longer is the DM pretty much given "carte blanche" to best make up what is happening behind the scenes. He is given a crapload of information to use in combat and also out of combat. Look at Paizo's run on the Dungeon magazine and you start to see the story trend, all leading to the adventure paths. However, imagine running Ravenloft using this format. No need to imagine, go look at Expedition to Castle Ravenloft. The 1e 32 page adventure has balloned to 200+ pages. If I recall correctly there was very little (maybe 3-4 additional situations) added to the content from the original.
4e kept the delve format. That can be counted as fortunate or unfortunate based on perspective. However, in 4e there was a difference. The system is particularly efficient in making those set-piece encounters memorable, and the delve format is particularly good for presenting information for those type of encounters. The real "problem", if all you give me is a hammer everything becomes a nail.
Due to presentation of the rules (combat & miniatures heavy, skill challenge mechanics that the designers didn't really know how to present well, forced combat encounters and poorly thought out initial adventures) everything became a set-piece encounter. Everything is presented in soul crushing detail, sometimes multiple times. Of course this is an exaggeration but IMO it is an apt one. Look no further than
Keep on the Shadowfell to see that the set-piece encounter was here to stay.
Now I do malign KotS quite a bit, particularly for its repetitive and sometimes obtuse nature (see the first encounter Kobold Brigands (p.16), and then look at Kobold Ambush(p.24). Isn't this pretty much the same encounter with one kobold added/swapped?). But I have used the module multiple times because there is still enough there that is salvageable to make a really fun adventure. The story of Sir Keegan is a very cool thread within the adventure but it is buried in a forced combat encounter already presented in Dungeon Delve format ("Sir Keegan’s behavior depends entirely on how the adventurers react to him. His first statement should be a strong hint that he is not meant to be treated as an adversary. His second statement makes it clear that if the PCs don’t convince him of their true intentions, Sir Keegan is prepared to attack them." Wait, what? Did any of the designers ever play D&D?). Put a monster in our path that gives us hints, even strong ones, and on his second breath threatens to put the beatdown on us and all you're going to get is "roll initiative." What were these guys thinking? Why does this happen? Mostly because the adventure trains the players to shoot first and not ask questions (see kobold ambush and every encounter in between). For the first half of the adventure the players have encountered the same monsters at least 3 times (dull, repetive and boring), each time in combat. Now you want them to change tack based on strong hints and threats? Please, get real!
Kalarel, the main villain is a cardboard cut-out mustache twirling villain and has little to no appearance except as a ritualist in the big end encounter. I don't even remember any mention of Kalarel, by any of the other creatures, throughout the adventure (but I might be mistaken as its been a while since I read the whole thing). Ninaran, the BBEG's henchwoman, is another non talking or interacting cardboard cut-out ("A quiet elf hunter who usually drinks alone, Ninaran is not interested in conversation"). What a bunch of wasted opportunities. The first module series of WotC adventures (H1-E3) had many of the same problems. The Scales of War series on Dungeon magazine suffered from a somewhat disjointed storyline that was at times difficult to reconcile as a series. The Living Forgotten Realms adventures suffered from a different problem they tried, very hard, to emulate what WotC had done in its presentation format, and do that in 28-32 pages for multiple levels. Some of the stories/plots for the first adventures were OK, and some were pretty terrible, but the real issue was time management. The DM for these adventures only has 4 hours to get to the finale. So a lot of these adventures were built on a common template/blueprint of 2-4 combat encounters + 1-2 skill challenges. Everything was quite mechanical, predictable, and at times forced (nope can't use rituals here because, "break linear adventure"). Thankfully, the adventures in Dungeon, and LFR got much better as time went on. There are some that are fantastically imaginative.
The truth of the matter is that a DM can make whatever he wants out of the adventures. To this day I don't think I've used an adventure exactly as presented. When I was running 3.x and during the whole time I've run 4e I've blatantly taken from adventures as needed, and convert adventures all the time. Many times on the fly. The most fun I've had in a while was when I converted (A1-A4) the slavelords adventures to 4e, and used some of the NPCs in very creative ways with hooks to the PCs. That thread is still playing out.
But experience has taught me how to do this effectively. What does a new, inexperienced DM, have to supplement that gap? I believe that is why adventures have so much fluff detail. To make the DMs job easier. Does it always work? I'm not sure. Sometimes too much detail can lead to straitjacketing (see Dragonlance for an example). The other side of the coin is that too little detail and the DM might feel cheated. Why am I paying $25 for this adventure that has minimal details?
So for publishers I think it is a rough balancing act.