Where the heart of the game really lies is in the situation and the players' choices in response (which in turn elicit responses from the environment, and so on back around).
What one really needs is a visualization of the circumstances, clearly enough conveyed to the players to inform their decisions. Game-mechanical formalisms are useful -- and thus needful -- only to the extent that they facilitate some interaction between the players and the world.
The resources in some games for "building" monsters and NPCs remind me of the design sequences for vehicles in the Striker miniatures rules (part of GDW's Traveller line) -- or such successors as Fire, Fusion and Steel or GURPS Vehicles.
That kind of thing makes some sense in a wargame (although vehicles ready to use would be more appreciated by many players). In the context of a role-playing game, such devices slip away into being basically a solitaire game for the GM. (It may be for players as well, but that is not really the matter at hand.)
The essential tool kit for improvisation is a stock of inspirations.
I get a lot of leverage out of the accumulation of "sense of place" as a campaign progresses. There's less need for detailed itemizations and maps when one can use reasonable inference (and perhaps a few dice-rolls) based on general knowledge. That such knowledge exists in the first place allows players also to learn and use it.
Also tremendously useful are fleshed-out NPCs. When the dynamics of relationships are in the game, I find that at least one answer to the question of "What happens next?" usually follows very naturally from what has happened so far.
Stock locales (e.g., maps of typical buildings) and characters (e.g., stats for typical soldiery, which may serve -- as in real life! -- once as a baron's levies and again as bandits) are another resource one tends to collect over the courses of campaigns.
Lists and Tables generally are nifty to have. Names of all sorts (people, places, magic weapons) can sometimes inspire creation of the named; an online "Vancian Spell Name Generator" comes to mind. In any case, evocative names can be -- YMMV, perhaps in inverse proportion to the number of apostrophes involved -- an aid not only to flavor but also to ease of play.
Among the extremely various material in the first-edition Dungeon Masters Guide is Appendix I: Dungeon Dressing (Miscellaneous Items and Points of Semi-Interest for Corridors and Unpopulated Areas or to Round out Otherwise Drab Places). That series of tables covers: General (mostly trash); Air; Air Currents; Odors; Unexplained Sounds and Weird Noises; Furnishing and Appointments, General; Religious Articles and Furnishings; Torture Chamber Furnishings; Magic-User Furnishings; General Description of Container Contents; Miscellaneous Utensils and Personal Items; Clothing and Footwear; Jewelry and Items Typically Bejewelled; Food and Drink; Condiments and Seasonings. Appendix K: Describing Magical Substances is a collection of descriptive words for Appearance/ Consistency, Transparency, Taste and/ or Odor; and Colors. The potential utility of a thesaurus may be evident.
Especially when designing dungeon levels, I like to start with a "brainstorming" session. I will sit down for a while and just write down whatever ideas for encounters and whatnot happen to come up, without getting into much detail. A key is not to analyze whether an idea is "good" or "bad" -- even for the setting (e.g., dungeon level) at hand -- but just to jot it down and quickly move on. I may have a target of, say, 50 entries in a given period.
What I don't use I often add to my "bits box" of scenario-building material, along with ideas that come to me in the course of daily life. The real world can provide a lot of ideas as well, and clippings from newspaper or 'Net may find their way into my notebook.
Genre fiction pretty often comes to mind, but it can be tricky to implement such inspirations. There is often difficulty in player familiarity, and sometimes things that seemed fresh enough at the first encounter have since become stale clichés. Then again, they may be "classics" worth revisiting in moderation. Sometimes the problem is just the opposite: players unfamiliar with the reference may not 'get' it.
How much players 'get' and 'get into' the game can also depend on compelling presentation. When I consider applications of the art of 'storytelling', plot -- normally the first requisite in literature -- falls by the wayside, for the discovery of what happens is my purpose in play. At any rate, a plotted scenario is probably the epitome of preparation, at the opposite extreme from "winging it".
The use of language is very important. Drawings, lighting, models, music, props, and more -- wherever one's inclination, talents and time lean in creative expression -- can make the difference between 'good' and 'excellent'.
All of the above strike me as more important "irons in the fire" than carefully calculating game-specific mechanical details. That tends to get in the way of keeping the game moving, which is in my experience a top priority.
Where I think the DM's job has really evolved is simply in a direction away from "winging it". Besides an increased emphasis on using canned adventures, I see a greater tendency to emulate those commercial products in "home brewed" scenarios. There has been a feedback loop of synergies, so that what players tend to expect of DMs -- and DMs of themselves -- has changed radically from the early years of D&D.
