I think you have enough dice when you can cast the spells used by most characters without rerolling, or roll all your attacks at once- including haste - with matching damage dice.
Therefore, I should have 5d4 for magic missile, at least 10d6 for fireball type spells or sneak attacks, 5d8 for searing light, holy smite, or damage rolls, etc. Once you get enough dice to properly handle almost any character class's abilities, then it might be a good idea to stop buying, unless you want to have expendable dice for loaning to other players.
At higher levels, this can get rather annoying. For example, spells like Horrid Wilting with caps at 20 or more d8s might get expensive. However, I think it takes forever when Hasted casters have to reroll a few dice for each spell they cast, and of course the laws of Murphy and gaming dictate that some of the dice are going to fall off the table and waste more time. Even people who don't really know what to go can finish a turn faster than a hasted wizard that needs a to reroll to finish each spell.
