4-6 would be my favorite. My best game had 5, and it was a perfect balance of enough people to get the job done and not so many that everyone couldn't be heard. The dynamic was great with that game.
Unfortunately, right now I've got extremes to either end. The "day" game currently has three players plus the DM. I DM that game about half the time, and I'm really having trouble. The going is slow, because there's more time for the players to get bogged down in little things. It's hard for me to gauge encounters, and I find XP and treasure more challenging. Also, I have a couple of players who like to agitate each other, so we spend a lot of time dealing with the good-natured bickering. I wish I had just one more person.
In the "night" game, I don't DM generally, but we have the opposite problem. Seven players plus the DM means that combat takes a long time, balancing encounters is still tricky, and it's hard to manage everyone.
Unfortunately it isn't possible to shuffle people about to average the numbers out.