Just give them a -1 on arcane spellcasting levels, it is not completely limiting but would hurt them quite a bit. Imagine this with 1st level characters, why would anyone want to play a wizard or especially sorcerer with this race -1 level of spellcasting and -2 charisma. Virtually no sorcerers would come from this race unless it was purely a roleplaying choice.