Actually, the situation sort of makes sense, although the other players' and DM's view on Charisma doesn't. Here's how it looks to me:
The other players thought your PC was unpleasant and, presumably, either overly sarcastic or bossy (or both). Hence, that high Charisma, since in their (flawed) perspective it didn't come from her personality, had to be based purely on appearance. Many groups I've encountered play Cha this way - since it represents a combination of good looks and charm, it must mean that one is extremely high if the other isn't. Since they found your character anything but charming, they assumed she was good looking instead. It's not a male/female thing, it's a pleasant/unpleasant thing.
Now, because they got this impression and expressed it, clearly annoying you in the process since it's not what you wanted from the character, you decided to play it up. Fair enough.
But remember: they
already thought your PC was a bossy, snippy witch, so when you played it up more, it made her, in their eyes, an
over the top, stereotypical version thereof. Maybe they're right, maybe you're right, but either way, that's their perception.
Unfortunately, they and the DM apparently think that, since your PC's personality has gotten more unpleasant (= less charasmatic, to their eyes), and she presumably hasn't spontaneously gotten better looking, then her overall Cha must have gone down.
Now, what they're missing is the 'third element' of the Cha score: strong sense of self and assertiveness. The more confident, domineering and perhaps even unpleasant a character becomes, the higher the Cha score goes because they simply enforce their strong wills on others. A mind flayer is charismatic, not because it's good looking to human eyes (one hopes

), not because it's a charming host, but because it exudes inhuman, unnatural self-assurance.
Call it confidence, comeliness and charm - when charm goes down, if Cha remains the same and comeliness doesn't change, confidence must be going up.
Personally, I'd go back to playing your character the way you orginally played her, which although not exactly one that the other players would want to hang out with, was obviously not "beyond the pale" like the way you started playing her to live up to the perceptions they already had.