It is due to a curse after all, so maybe its not entirely up to the player to act fully CHA 1, it could be the curse causes others to see and treat him as CHA 1. Essentially, his opinions have no weight, his presence is not noted or desired, his actions are not appreciated or acknowledged...