For my 20th level rogue/fighter I had enough that I could roll once.  So 2 D20s (in case of advantage/disadvantage) plus D8s, an extra die or two (custom weapons that did extra damage) per hit plus sneak attack dice.
The trick to keeping track was that I use a clear plastic sheet protector for my character sheet that I recorded AC hit on one row and damage on the next for each color.  Some dice (green in sparkly) were if I was action surging, and I'd record crits with a "C" So I had something like:
| Sneak | Black | Blue | White | Green | Sparkle | 
|  | 16 | 23 | C |  |  | 
| 24 | 20 | 21 | 30 |  |  | 
 
But that wasn't the worst, I've done this for a long time and had a 3.5 character (with animal) that needed close to 60 dice when he was retired.  I roll/add while everybody else is taking their turn so my turn takes less than a minute.  Does a 16 hit? No?  What about a 23? Ok I do ____ damage.
So yes, I have a really big dice bag.  Something like
View attachment 92192