The problem is: I can design and brainstorm during any spare thought. So, it's not really a tangible length of time. I could be at work, driving, playing poker, sleeping, ect...and still come up with some reasonably fun ideas--including dungeon design and story arcs.
I actually spend far more time on design and ideas...but not physically. Writing stats, drawing dungeons and towns takes up physical time which detracts from other aspects of my life. So, the physicallity of the work involved in, stats and maps, is more consuming and invovles more of a commitment. 'Course, I'm usually still brainstorming while I'm doing the the stats and maps.
I gotta go with the brainstorming--with a more realistic nod towards the stats. If you had asked: what aspect of game prep most interferes with your life? I'd have to say writing up the stats.