Somewhat. Many monsters are just wildly off in how challenging they are. A will-o-wisp will ravage a 6th level party... 9HD, +16 melee touch for 2d8, TOUCH AC 29 before you factor in their natural invisibility, +13 to initiative, immunity to all spells except maze and magic missile... So lets see, odds are the wizard probabnly has 3 magic missiles that wont kill it, he drops in a couple of rounds, and then everyone hopes for a natural 20 to hit?
NPC's are anotgher bad error on CR's part, particularly with nonassociated class levels on creatures with racial hit die. Even with base NPC's, you cant tell me that a 9th level cleric, a 9th level druid (with its animal companion that doesnt give exp), a 9th level warlock and a 12th level commoner are anywhere near equal to one another on the power curve. Some PC classes just dont hold up well... the warlock for instance, a class that benefits from doing the same thing over and over. However, most NPC's dont get into more than 1 fight per day, so the warlock is just a nuisance compared to a more powerful class. Add in that non-caster NPC's are usually complete pushovers compared to monsters of an equal CR (yet give far more treasure), and something is definately wrong.
I think my favorite is the planetar, a CR 16 creature with a crapton of special abilities that ALSO casts spells as a 17th level cleric. You'll note that a CR 16 cleric doesnt even cast spells as a 17th level cleric. Furthermore, the "good guys" are almost always listed at a CR rating lower than they would be if they were evil. COmpare a solar to a pit fiend with a few more hit die tacked on.
The CR system has some good points, but also some noticable flaws, so I think it should be avoided at all costs when actual mechanics are involved (knight's challenge and the abomination known as the truenamer). As a guideline, it works close enough for government work.