Regardless of the game system I use, it always seems to take me 3-5 hours of prep per hour of play time that I want to have. And for me at least, it seems to require nearly that much even if I'm running a prepared published module.
To only spend four hours prepping a session is astonishing to me. Normally I'd spend 10-20, and I'd only get below that if I had a sandbox which I'd invested in that was returning on the investment. And most of that would not be working out mechanics. Most of that would be working out the fiction and the setting, figuring out how to animate the NPC, making maps, or doing research. It took me just as long to prep for 1e AD&D with its super simple stat blocks as it does for 3e D&D, because what I lose in lost time churning out 5 line stat blocks compared to 1 line stat blocks, I make up for knowing that I have a tool set that can handle interaction with the environment instead of needing to smith out location by location rules. Read a 1e AD&D module some time and take note how much of the text is spent creating on the spot rules for the specific environment of the room.
That and word processors are a God send.
My advice to you regardless of the edition would be don't roll up an NPC rogue or stick a template on them. Give the character the attributes you want. If you are 1 or 2 'plusses' off of what a rigorous check of the math would yield you, so what? There is only a small chance such a small difference will matter to the die roll anyway, and most the work you're putting into getting the skill points to come out right is wasted anyway because you won't make all or most of those skill checks in game anyway.
Reuse stat blocks. Once you've created one buccaneer stat block, you don't need to create another one. Flesh out important NPCs only as you need to. By the time the campaign has gone 50 weeks, you'll rarely need a stat block for an NPC that isn't just cut and paste from another one and can't be tweaked on the fly if you need to.
In my experience most DMs don't prep enough (including me). But also in my experience most DMs are prepping the wrong stuff.
Over the winter break I ran a game of Mouse Guard - six hours of prep per hour of play and I don't think I've ever been as frustrated with the inflexibility and bad math of a system since I played RIFTS that one time in college.