I think the only time the dice size matters is when you have that one individual who never misses and the dice are always hot yet they're so small or unreadable (looking at you cthulu dice) that no one else can tell whats actually being rolled. And then it's more of a social contract to use different dice than an actual requirement.
Everyone says that, but I can read my cthulhu dice from across the table. Of course, no one would ever accuse my rolling of being hot.