Now, that is what I do generally. In certain campaigns, like the city-based one I'm running now, I handle tasks in town via a town task system that breaks the basic tasks an adventurer might want to do in town into a mechanic that resolves quickly while providing some meaningful choices and outcomes. Much of these are based on downtime activities, more or less. I set some kind of limit on the number of tasks that can be performed per PC per day and make sure that Team Villain's agenda is on a timer, too. So it's like rounds of combat and just as quickly resolved. Of course, spending too much time on this kind of stuff, while beneficial, just means that the antagonists get further along with their plans and are that much stronger when the PCs face them. (The town tasks are also not worth XP whereas pursuing adventure and its inherent challenges is!)