Actually, I wouldn't encourage you to play him quiet and sullen.
Think about the most socially awkward guy you can. The one who gets too close for your personal space comfort. Says things inappropriate for conversation/context of the interaction. Isn't aware of other people's reactions (such as continuing to talk after someone has sent signals they want to leave). Talks too loud, and has no internal editor. He likely also interrupt with non-topical questions.
That's low charisma, right there.
To avoid getting on the other players nerves, direct your annoying behavior towards NPCs. For instance, just as you meet your patron, say loudly as she is almost out of earshot, "Wow did you see how fat she was?"
Basically, the comments that everyone makes around the table out of character? Have him make them in character. Have him toss out lame one-liners, instead of good ones, or do them before (or far too long after) the appropriate time to throw them.
Try to do it in a way that's funny to the Players, but annoying in character.