There's two types of prep time.
One is the pre-campaign prep, where you build the world, history, cultures, maps, etc., etc. and at least try to come up with some stories and-or adventures to shoehorn in there somewhere. This is where all the time goes; I spent well over a year designing my current campaign before dropping the puck...I've learned that the more time spent on this part the less time you need to spend later, as so much of the campaign can kinda run itself.
The other is the during-campaign prep., which if you've done the pre-prep right should usually be minimal. The between-sessions time is now spent record-keeping and fine-tuning...and reading modules to see if they're worth bothering with.
I have no problem winging it, except I've occasionally winged myself into a corner (beer was probably involved too); and I've run entire and quite successful adventures off a single sheet of scratch notes and diagrams.
A good example is the one I just started this evening. Last session one of the PC Clerics got quested by Corellon: I had to dream up a quest on the spot (I had no idea this was coming, thank you dice) so I muttered something about "start at the tower of the halo" and left it at that; and they went on dividing their treasury from the prevsious adventure.
During the week I spent an hour scribbling down a page of quick notes on what a "tower of the halo" might represent - and that'll be the printed part of the module right there, along with a rough map - so that tonight when the Cleric went asking bards and sages what this was all about I had something (only very slightly) more concrete to talk about. But at least I now know what and where the "halo" is, and what it does, and what defends it, and why, etc....even if the PCs still don't.
In the past I've sat down and prepped sessions to the walls only to have the PCs throw it all in a handbasket within the first ten minutes by doing something unexpected, leaving me winging it anyway. Now I find myself winging a surprising amount of what goes on, unless I'm running a canned module.
Lan-"flap flap flap"-efan