I view the Sorcerer as having a magical bloodline. In other words, magical "technologies" have engineered their bodies to perform magical tasks. The Sorcerers dont need to understand these technologies, but need experience to pilot their bodies under stress. Because Sorcerers literally use their own body as the source of their magical powers, it is one of the few character concepts where using Constitution as a spellcasting ability makes sense.
Warlock is inconsistent. They make a bargain with a magical creature. Maybe Charisma relates to skillfully navigating the personal relationship with this magical creature. But the rest of the flavor is inconsistent. Sometimes the Warlock is more like a Wizard where the Warlock learns how to do magic while the creature is a mentor. Sometimes the Warlock is more like a Sorcerer where the creature magically transforms the Warlock to do magic, so the Warlock doesnt really understand the magic and is "cheating" when being able to do magic. In this context, the abilities of Intelligence or Constitution could make sense.
If the Warlock is Charisma, maybe the magical creature did magically transform the Warlock, but the new magical nature requires the personality of the Warlock to guide this magic. Maybe while leveling, the Warlock needs to talk the magical creature into giving "upgrades" to expand the new magical nature to do more magical features.
In any case the flavor of the Warlock is ambiguous.
The more I think about it, to me a 5.5E would allow a Warlock to choose either Intelligence or Charisma as their main Ability Score (reflecting that in playtest material Warlocks were Intelligence, which makes sense; the fact that on 5E's arrival there was only class at all concerned about Intelligence besides skills and saves is baffling to me).
Then, for a 6E, Warlocks as you suggested would have a variety of different main scores depending on their patron, to reflect that as a class they 'break the rules' in many ways and have more complicated and difficult progression. One key rule however would be that Dexterity could not be the core ASI and that Hexblade in particular would need to be carefully chosen to make the subclass less prone to being chosen purely for multi-class 1 level-up shennanigans.