It is due to a curse after all, so maybe its not entirely up to the player to act fully CHA 1, it could be the curse causes others to see and treat him as CHA 1. Essentially, his opinions have no weight, his presence is not noted or desired, his actions are not appreciated or acknowledged.
Maybe a fun roleplaying aspect might be that other party members constantly forget the he is there, or maybe even who he is, accidently leaving him behind, etc. And perhaps things he says are dismissed out of hand regardless of their content by the party. Like the PC gives his idea of the plan, but the others are like "Shut up, Ted, that's dumb. Hey I just had an Idea [says Ted's idea without realizing its Teds Idea]." That sort of thing.