I think this kind of question really depends on what your players are like. I get restless sitting down and just listening for long periods on end (for instance, while other players are running through their turns) so I often fiddle, like playing with dice or knitting, but I'm always listening...