You gain a more variable - and thus, both more realistic and believable - set of characters.
Your second sentence disagrees with your first, in that you are saying I shouldn't roll unless everyone else actively wants to. That implies (to the point of almost outright saying) that one non-roller...