I think it highly depends upon system. For the two editions of D&D I'm most familiar with (3rd and 4th,) the sweet spot seems to be 4-6. That being said, I ran a very successful 3rd Edition campaign which only included 2 players; it helped that one player was a druid (and thus had an animal companion) and the other had a NPC cohort... so, really, I suppose that mimicked having a 4 player group. The most players I ever had for a 3rd Edition campaign was 11; I handled that by breaking the game in half and having a second DM help with a lot of things. It worked pretty well, and it allowed for the group to be split for some pretty interesting story telling scenarios in which not all of the group was together.
Most of the 4th Edition games I've run have had 5 players. That is the recommended amount, and it does seem to be best for 4th. Four players works well too, but it depends upon what roles are covered. Larger groups worked well in some of the games I ran too, but I noticed that doubling up on certain roles really throws the system assumptions out of whack. For example, I was a player in a group of 6 that had 2 Warlords; no controller; one fighter (defender) who could put out damage similar to a striker, and three strikers. With the extra attacks and bonuses that two warlords could throw around, the enemy rarely survived the first round. The group also had plenty of healing. In contrast, I DMed for a group that had an over-abundance of defenders; while they had great defenses, they often had a hard time finishing a fight... they had a lot of staying power, but very little in the way of offense, and -since the increased group size meant more monsters per encounter- they were often outgunned; encounters were even more grindy than usual.
My experience with older editions of D&D is very limited, but (thus far) they have seemed a little more flexible when it comes to party size.
For non D&D games...
I haven't noticed any significant difference in how the system works when GMing GURPS for groups ranging from one player to eight players. The challenges which the group is best equipped for certainly changes depending upon the group size. However, I haven't felt especially pressured to change my method of building encounters. Also, since combat tends to go quickly -due to combat being a bit more lethal than D&D tends to me- it takes a lot of moving pieces before I feel overwhelmed. It's rare to have grind, an the flexibility of the system allows for non-combat encounters to be handled in a variety of ways.
It's been a while, but I seem to remember Rifts getting a bit crazy with a large group. I feel it was a difficult system to handle when more than 4 players were involved. I think a large part of that may have been due to the clumsiness of the system in general though. Also, I was far less experienced with rpgs in general back when Rifts was one of my primary games.
edit: my ideal size depends upon what I'm running. For D&D, I'd say 4-6. If I'm running GURPS Vietnam, and the players are members of an infantry platoon, I'd feel comfortable with a large group. If I'm running GURPS Dungeon Fantasy, I'd feel most comfortable with anywhere between 1-5.
