Optimal number? It simply differs in so many ways, to even games and players who don't care for classes.
I don't know the correct amount for any game, but here's how I break it down for D&D at least.
Pre-d20: There are 3 primary classes with 1 NPC class tacked on later as a PC class. Each of the initial 3 has an entire game system devoted to it: the combat system, the magic system, and the clerical system. Each system overlaps sufficiently to make working together beneficial to players of each class. But then there are many sub-classes, classes which fall more or less in one of the core classes and systems. Each sub-class has perhaps a little bit more scope than the core or maybe a little bit less. For some, part of the initial system-defined core class scope is left out, some may have more included, some may have the sub-class overlapping into another core class system. Some may have all three. It's very flexible. Psionics, when included, trades in powers and out others by each core class rather than being a class or class system in and of itself.
3e d20: There are 11 core classes with 2-3 optional psionic classes. Many more core classes would be constructed later, but psionics began the idea of new classes falling under different systems. Notably, all classes, regardless of definition, are geared for equal participation and effectiveness in a combat game, although a lot of activity not normally performed in combat is still included. Beyond the core classes, prestige classes were created (something perhaps only the Bard class was before). The number of prestige classes published is far more than the number of core ones. And while optimal numbers may differ by group, perhaps this is a case where too many could be seen as an impediment to running the game.
4e d20: Back to 4 core classes: controller, defender, leader, striker. They remain as combat classes, but now focus on the scope of combat each class excels at. Sub-classes are now called classes and a large number has been printed with varying themes built. with thematic combat powers the game retains something of the original classes: martial, arcane, and divine. Primal was added and so too was psionic. Being thematic, these differ mainly in description than game play, but I believe they do tend to group along common configurations of powers outside of the roles/classes (but I'm not sure here).
It's hard to say how many classes are optimal in any of the above cases. Maybe more should be added, maybe fewer? It's really more of a group and DM decision, of course. I do believe there is an upper limit to what anyone can currently handle in terms of options. I mean there's something to more options than can be played in a lifetime, not to mention having more options than can be read in a lifetime. Single digits are probably best in the end, but then I would allow sub-class variations either published or customized by the player.