That's one of my big problems with the system. I ended up with three sets of dice for my group (which still wasn't enough) and then realized the system works best with the minimum amount of rolling.
Except for combat, which requires constant rolling and gets super bogged down when it's supposed...