In general, I usually look for about 2/3 damage for a radius 10' AoE, which I assume will hit about 2 or 3 targets. That way it's an obvious loss against one target, pretty much break even at 2 (since focus fire is generally a better tactic, even hitting 2 targets needs to be higher than 100% of single target damage), and an obviously better choice against 3.
Bigger AoEs should do less, but no less than 50% of a comparable single-target. Single targets should also tend to have better riders for their spell level.