I would say worry about your main gods (ie life, death, peace, war, nature, elemental gods, etc), then fill in the slightly more "niche" ones as you need them. A dozen or so 'generic catch-all' gods, a handful of racial ones, and a few specific/silly ones.
I'd say you have too many when you...