The one thing I'd add to this discussion:
While the CON penalty might balance the INT/CHA boosts nicely on paper (since as has been pointed out, a -4 CON on a d4 HD really hurts), think about what'll happen once you get to higher levels, and people have access to +CON and +INT/+CHA items. If a player knows his character will always have access to these items (say, if he's being generated purely for a high-level campaign), the balance doesn't work.
Let's say we're using a point-buy system, and Character A (stock human) starts with INT 16, CON 12. Character B (your elf) starts with INT 18, CON 8 for the same number of points expended (and of course he'll also have +2 DEX, +2 CHA, and -2 STR, a net gain for a caster). Okay, at low levels, character B will be crippled. And if he uses his level-based stat boosts to raise CON by 2, the human could do the same for his INT. But what if B waits until they both have access to a +4 CON item and +6 INT item?
A would have INT 22 (plus level boosts), CON 16.
B would have INT 24 (plus level boosts), CON 12.
Basically, after a certain point, extra CON is only really useful for a caster if you EXPECT to be hit; your spells give enough AC/miss chance/whatever to keep you safe most of the time. So maybe character A won't bother with the CON item, but nothing he does will let him catch B in INT.
This also applies in a high-powered campaign. Say, for instance, a 32-point buy system; it's not hard to have a CON of 14 under that system, and subtracting 4 still gives you an "average" score.
Now, I'm not saying that caster-oriented races would be a bad thing; a little bit of pigeonholing can work to your advantage by providing a baseline to construct a culture around. But personally, I'd do it some way other than raising casting stats. (And remember that INT+2 basically duplicates the Human racial skill boost.)
For instance, if you're trying to boost the ability of the race in regards to arcane casting, give some other boost. For instance, pick one or two of the following:
> +1 to arcane spell DC (which mimics the INT and CHA boosts, but without the bonus spells or skill benefits); maybe limit it to a specific school.
> Any caster-level-dependent benefits of the spells treat the caster as if he were a level higher. (A fireball by a level 5 Wizard would do 6d6 damage). All normal caps still apply.
> Certain spells are always treated as known by the caster. (IMC, we did this for Gnomes, giving them their three cantrips for free and letting them cast two extra cantrips per day, instead of each being 1/day.)
> Let them refresh their spell repertoire with only 4 hours of rest instead of 8.
> Give them the ability to counterspell more effectively (however you want to do this)
> Free metamagic. For instance, let them empower any spell 3/day without raising the spell's spell level, with the Sorcerer's increase in casting time.
This lets you boost their casting ability a little without requiring such a huge drawback; you might only need CON -2, and that's a lot less limiting on a character's construction.
Now, EyeontheMountain pointed out that races with tons of ability adjustments are iffy, but I've done it before, mostly successfully. We had an ECL+0 Half-Ogre in our campaign that was (STR +4, DEX -2, CON +2, INT -2, WIS -2, CHA -4), and while it was of course skewed towards melee combat, it fit into the world remarkably well.
