I believe Bards should be CON-based because, well, you know...
But on a more serious note, I think there's an argument to be made for warlocks to be CON-based. Like
@Minigiant said, patrons grant powers in different ways, and in the PHB, under the Great Old One,
The Great Old One might be unaware of your existence or entirely indifferent to you, but the secrets you have learned allow you to draw your magic from it.
so there's a possibility that patrons might not even know their power is somehow being siphoned, or they decide to bestow some of their power against their beneficiary's better interests.With a little headcanoning from me, I think that possessing magic powers (particularly arcane ones) comes with a burden. For example, wizards use their intellect to safely handle their magic, bards
seduce persuade the Weave to do things they want, and sorcerers force the Weave to do their bidding through their innate magic. (Pure) Warlocks come across the power "illegitimately" without the proper training and/or innate nature, and their meat-sack bodies have to be strong enough to withstand the power coming from this foreign agent. The sturdier they are, the more control they have over it.
Mechanically speaking it also gives Warlocks more versatility in being a frontline fighter, and could cut down on short rest spam.