The inability to lead does not ever mean there is a willingness to follow. Leadership, as it relates to others' desire to follow you, is indeed based on CHA. Just look at the leadership feat.
Your low CHA might simply mean that you want to lead but are way too overbearing, blatant, mean...