I rule that there are many factors that can alter charisma, and that yes, physical beauty is one of them. For instance (and this is only an example) consider the following list:
Beautiful +10 to charisma
Butt-Ugly - 10 to charisma
Good speaker +5 to charisma
Bad speaker -5 to charisma
Good hygiene +3 to charisma
Bad hygiene -7 to charisma
Graceful +2 to charisma
Clumsy -1 to charisma
Witty +3 to charisma
Slow to speak -5 to charisma
You get the idea. All of these things and more can contribute to a PC's personal magnetism. And unlike the chart I gave as an example, there are degrees of all these qualities, as well as the many I didn't list. Usually, if a PC has a particularly high or low charisma, I let the player decide the reason for it. Sometimes it makes for high charisma PCs who are physically stunning, sometimes it makes for high-cha PCs who are normal-looking but powerful speakers.