I do on occasion (when playing) play with phone (checking emails and such), stack/roll dice...etc, but it's not because I'm bored. When my character isn't involved in a scene for whatever reason I have a need to fidget. I can't just sit there.
When I'm DMing it doesn't bother me as I know that...