First, I should note that EL is supposed to measure something OTHER than combat ability - total character effectiveness. For example, having a phylactery does not make a lich more effective in combat, but it is definitely worth EL.
Second, as noted earlier, monster HD are unequal, and this has been expressed in 3.5 rules for increasing CR as HD increases.
In particular, dragon and outsiders are solid.
If there were a character class that gave NOTHING but:
3 good saves, full BAB, d12 HP, and 6 skill points (or d8 HP and 8 skill points for outsiders, and decent skill access), it'd be worth taking 1 or 2 levels of even without any special abilities.
Second, it's not always true that EL is not worth it. Generally, and special abilities like spell likes or supernatural abilities that do NOT SCALE BY LEVEL, make an LA race not worth it. LA deriving from spell like abilities, or flat-DC effects, or effects with DCs that increase by HD but not by class levels also, generally become worthless. Abilities that do scale MAY be worth it.
The biggest notable issue in LA is that it hurts spellcasting classes far more than non-casters, like fighters and rogues. A well designed fighter/rogue (or fighter-rogue) can benefit from the LA class if the abilities mesh well. For example, a pixie rogue/sniper...
The other issue is that most of the comparisons I've seen pit low level LA creatures against low-mid level characters. However, this does not take into account the fact that the base classes are heavily front loaded. Thus, fighter/rogues with LA don't really come into their own until they have at least 4 to 8 class levels.
Finally, LA derived from stat increases and other stackable bonuses can be worth it - but it depends how much. At high levels, half dragon can be decent. +8 STR, +2 Con, Int, Cha, +4 natural armor, possible winged flight... For 3 LA, is probably fair. (Note that I'm completely ignoring the breath weapon, which is worthless beyond level 5). Half dragon, as a template, also lets you keep the base race bonuses, like human. Classes like Aasimar are NOT worth it, however. They receive 4 stat points for one level and lose base race bonuses... In our game, we made aasimar a template (so you get the human bonuses AND the aasimar bonuses for 1 LA), and it's still questionable whether it's worth it. (For that matter, anyone who thinks the elf bonuses are even vaguely comparable to the human bonuses has serious issues...)
The real problem with LA is that the specific LA values given are often just way off base.
I'm not sure the fact that LA races make poor spellcasters is a bad thing... it does mean that the most magical races (in terms of learned or acquired magic, rather than innate magic) are the ones with lowest hit dice, and that's important for the game genre. It just means that no one will ever actually play a giant=priest (do you know what the LA of the cloud giant priest in the book is?!?!?!)