Matching up with what most of the previous posters have found...
Our group has had anywhere from 3-6 PCs at the table most nights and the numbers have seem to work pretty well. But we all get along and the DMs have seemed to managed things enough so that no one has felt left out. So that you don't have to cancel all the time, at least 4 or 5 is a good minimum number. For us, the PCs of the absent players just ghost along with the party unless absolutely needed (in a dungeon crawl) or hang out in their room at the Inn (if in a city). Of course if something particularly vital is coming up, then we'll postpone.
One of my all time favorite campaigns was a series of dungeon crawls that had 8-20 PCs on any given night. Some of the players had OD&D, some B/X, and others AD&D, and the major milestones for a character included making it past the first night, and making it to 2nd level. But we all knew what to expect and somehow the DM had the gift that it was great to just be in the room, even if you didn't get to do much of anything that night.