Therein lies what I see as the essential oddity of the Favored Soul. Better armor, but no hit points to go with it. That being said, even though it is displeasingly messy, I think it helps balance against the PHB subclasses. (Well, assuming you ignore the Elemental Affinity errata and let them use it on all damage rolls related to a spell.)
Well, the Favored Soul averages two or three hp less per level than a fighter. If I were going to role one up and was looking just for numbers, I'd pick a Hill Dwarf for the extra one hp per level, and then get the Tough feat once I hit fourth level. That would roughly get a Favored Soul on par with a traditional warrior.