It also really depends what you want to use the character for. If you want to be the "pure caster", then yes, any ECL hurts you. But, not everyone is hung up on always having the highest level of spell.
It's like the debate over whether your Sorcerer X-2/Rogue 2 is better than a Sorcerer X; as a caster, the pure-class is stronger, but the one who spent a couple levels on Rogue has other benefits. The Sorcerer/Rogue mix is a lot like a Bard, and I know some people who are perfectly happy playing Bards; they'll never be the best casters in the group, but that's not a problem for them.
Monster levels simply replace the Fighter/Rogue type of levels. I've only played one campaign with a monster PC. His name was "Tweety" (nickname, not actual name), and he was a Giant Owl Psion. (Quick show of hands: how many people remember that Giant Owls have INT 10? As a Psion he could cast without hands, material components, or words.) He took Leadership, and his cohort was a Cleric/Rogue who rode on his back. Everyone assumed Tweety was just an exotic mount. Of course, once I hit Psion 8 and got Polymorph Self, things got even more confusing.
Anyway, Tweety wasn't really a pure caster; he was like a Ranger 15, but I got that by mixing seven or eight levels of Psion with a half-dozen "levels" of Giant Owl. It was just a different way of making a hybrid class. It still made for an entertaining character, though.