Warlocks that focus on Con and Charisma (which are a lot of them) have the same problem.
And the answer is the same for both of them: Put a some stat points in Str/Con and invest in heavy armor.
Or, you could go Genasi Ranger, build Strength and Int, backstory from Thay (I have a group of Freedom Fighters in mind I am starting to build a campaign for) and essentially dump Con AND Dex and keep the light armor while also getting a +1 to Will via 12 Wisdom. The number of surges will be low, as will Initiative Mod, but very workable. I have one built already for LFR games like that I may try next.