Actually, you lose two caster levels; you have to have proficiency with all martial weapons, which requires you to take at least one level of fighter, barbarian, ranger, or paladin. I'd probably take the ranger, though I suspect the barbarian is a superior munchkin option.