While I like the idea of 1 charisma meaning that you are the ultimate wall flower (you make so little impact, negative or positive, that you might as well be invisible), I think that would be boring to play. I think he reason I would go with physically repulsive, even if that isn't based on the full meaning and utility of charisma, it at least has the potential for being interesting, for having plot impact. If the player is just a robot, totally flat effect, incapable of expressing emotion...that could be boring for everyone.
One way I could see it working is that the player is instructed to play as he normally does. He still has his INT and WIS. INSIDE he is the same, CHA is how you impact other sentient beings. But no matter how he interacts, it just has no impact. Maybe the curse is that he is "locked inside his own head" like a daytime drama representation of someone with extreme autism. The party, who know him before and would go to great lengths to draw him out and interact with him will at least be able to communicate and work enough to get basic concepts accross, but to every NPC, there is nothing there to them.
What I like about this is that the play need not role play the rain main or a try to role play a brain-dead automaton, which can be boring, awkward, or even offensive. Instead, he can play as normal, but the results of any social interactions just are not what he intends. He is ignored. He is picked on. People get offended for seemingly no reason. People feel threatened by this mentally defective but physically (or magically) powerful adventurer and believe he should not be allowed to roam free. People accuse the party of taking advantage of him.
This would be FRUSTERATING for the player, but in a way that remains interesting and impactful. It would not be boring (if the DM does his or her job well) and it would not devolve into silly "me so ugly" antics. The player--and so the character--would have a strong incentive to find a way to lift the curse but not feel like he or she has no agency in the meantime.