I think this is really the heart of the old-school/new-school divide, at least as I see it. Around the time of 2e the focus of D&D shifted from dungeon/hexcrawling to telling a coherent, epic story starring the characters. I read a good essay somewhere about how this started with the Dragonlance campaign, which let the players play the characters from the novels-- including, for the characters who get killed off in the novels, knowing exactly when, where, and how your PC will die.
So if your focus is on telling coherent stories and making the mechanics of the game fit them, then yes, nicely balanced encounters and no random monsters make sense. But if your focus is on letting the dice fall how they may and having stories arise from the game, then that element of randomness, balanced by player skill, makes more sense.
In both cases, the game tells a story. In old-school gaming, the story arises from the situations the game creates. In new-school gaming, the situations of the game are created to fit a pre-existing story. I personally love both styles of play, and think both can be taken to dangerous extremes.
I think most of the D&D games that I have ever played in a third of a century have fallen into a middle ground. Even very early pre-1E games often had long term campaign hooks and re-occurring villains and plotlines. That didn't start with Dragonlance (although playing PCs of characters in a novel and being railroaded into a dozen or so specific modules did).
The DM tends to have some ideas for a pre-existing story(ies) and leaves hooks around for the players to work with, but even the DM changes his ideas and his monsters and encounters, sometimes from session to session. The DM might be running a given module and railroading the players a little with it, but the DM decides some portion of the directional options and the players decide some portion. I don't recall campaigns where either the DM forced the game to always go in one direction, or one where the players mostly decided to ignore the DM and go off in whatever direction they wanted. It's typically been a middle ground.
To me, the difference between old school gaming and new school gaming is one of rules light versus rules heavy and the resulting DM/player style differences because of that.
In old school gaming, the game system had a lot of charts. There were a lot of different tables and charts on how things should work. But, some of these charts were things like random monsters or random treasure or adjusting to hit based on armor type. They were unnecessary and often dropped in subsequent versions. The game seemed a bit more complex than it actually was and most of the real complexity was due to inferior game mechanics where the industry hadn't yet fine tuned some of those mechanics. As an example, early versions of Turn Undead had a complex chart, 3E had a much simpler chart, and 4E got rid of the chart completely.
But in old school gaming, although the game system was mechanically a bit more clunky with more table lookup, the game system didn't have as many rules. The 1E PHB is a lot thinner than subsequent PHBs. The game could be played with or without grids. The DM tended to have more power with regard to adjudication because there were fewer rules telling everyone how a given unusual circumstance could be handled. And even 1E was split off into a Basic version of the game system to make it even less rules intensive. The game felt different from table to table because DMs handled a lot of things differently because either there were no rules on the subject, or the rules were somewhat vague.
As D&D has matured, the number of rules and clarity of rules has increased. At the same time, many of those rules were simplified or became more efficient. As each edition of the game system matures, there is even more rules bloat as more and more splat books are released. The rules are more efficient, but there are more of them, so the complexity is constantly high.
But, that didn't happen in 1E. It took 2 years for 1E to release the PHB, the DMG, and the MM. Rules bloat wasn't the norm. There was no Internet. DMs couldn't just download pre-made adventures, they built them themselves or went to a hobby shop and bought them (and in that case, most DMs had limited choice and many different groups played the same modules).
So I think that a lot of the "old school feels more free form and more reactive to the situation created than new school" ideas stem from the fact that there were fewer rules, more on the fly adjudication on the part of the DM, and less access to pre-published material. The DM did a lot more "seat of the pants" decision making because he didn't have as many tools and rules to handle adjudication and world building. The work required to DM the game was so much more intensive that the DM often had less of the important parts of his world fleshed out and more unimportant things fleshed out. He had to on the fly adapt to the player situations because it did take more effort to create monsters and build his world and adventures. Plus, DMs were naive in the earlier days. They would create drinking charts (Constitution checks for consuming alcohol), NPC reaction charts, and a wide variety of other house rules and charts and such because they thought it was important and because the game system wasn't mature enough and DMs were not experienced enough to not know that this was unnecessary.
In latter versions of the game, a lot of this DM legwork was handled for them, thus freeing a DM up to work on more important game elements and fleshing out pre-existing stories more than previously, thus making them feel more like pre-existing stories.
But, I've never been in a game that I can remember that the pre-existing story wasn't often sidetracked with side quests and new situations because the game does evolve from session one to session N. The DM isn't actually telling a story, the DM and players together are telling a story and that story is always evolving and nothing like the pre-existing story that the DM originally set out to tell.