Did you lose Cha permenantly of temporarily?
A lot of adjudicating the Leadership feat requires some common sense and fudging to work well IMO.
If it is just a temporary thing and you treat your cohort well, I would expect him to stick with you, for a little while at least. He is still a full-blooded NPC with his own ambitions and personal relationships he cares about, not just an automaton controlled by the rules.
If your effective leadership score "permenantly" drops, this can happen for a number of reasons, then you may find your cohort or a few of your followers start to drift away... "I have some important family business to take care of at home, sir. Maybe I will catch up with you in a few months."
From the within the campaign logic, the cohort may have decided to ignore personal matters while you were still such the awe-inspiring hero in his eyes. With your leadership score drop, he decides some of his personal business should not wait.