I think CR has been a step forward from the much vaguer situations in 1e or 2e, but it should not be treated as a hard and fast rule.
That's fair, and very well-chosen, because it is a step forward from those editions. And a lateral move from 3e, and a big step back from 4e.
Which is so frequently the case with everything 5e has done as a Compromise/Big-Tent edition trying to pull the high points from each prior ed, that we really should be used to accepting it, by now, 5 years in.
I mean, 3e introduced CR, and 4e improved on it substantially, but 5e,
had to call back the TSR eds, as well. It couldn't make encounter building into a detailed optimization mini-game counter to the players build-optimization mini-game like in 3e, or a neatly-balanced exercise in set-piece combat 'scene' choreography like 4e. It had to bring us some of the uncertainty and DM/player 'skill' of the olden days, too.
In other words, I fully agree with you that producing fair and interesting challenges still remains more of an art than a science.
It is, indeed, once a gain, primarily an art - a matter of feel, experience, improvisation, and creativity. Which is both a good deal of fun, and takes a good deal of skill/talent/energy to get the most of out of.
So, yes, the following are, once again, true, and advisedly/intentionally so:
1) CR is often wrong. Assigning CR in the first place is an artform, and it's often wrong in published material by +/-1 or in some cases even +/-2.
'Wrong' is the er, wrong? word for it. YYMV, maybe? CRs in published materials seem to follow a formula, that formula, thanks to BA, doesn't stand up to the linear-distribution randomness of the d20, nor party composition nor optimization, since balance on the player side is also far more varied for similar reasons.
2) Even when CR is not wrong, the range covered by a single CR is fairly large.
It maps to level, still. While BA means a party can functionally engage CRs far above and below them, it's still centered around a lone same-CR critter as a meaningful little challenge.
3) Every campaign and every party is different in its assumptions and in its composition, and as such different challenges will be of different difficulty to different groups.
Yes choice of class, build choices, pacing, and many other factors radically distort balance among classes and vs encounters. It's inevitable given the mandates of 5e design. You simply can't give some classed no meaningful daily resources, and others game-changing ones, and give players any freedom in the 'length' of their day
and expect any set of encounter guidelines to hold up!
4) CR never takes into account tactics. In particular, the terrain that a fight takes place in massively alters the effective CR of something.
Well, it can, if the DM sets it up that way and runs to that level of granularity.
In this case, though, I think 5e CR deserves a free pass: the game assumes TotM, TotM doesn't support highly granular tactics.
5) CR never takes into account synergy. Synergy happens when two monsters are more than the sum of their parts, because each is able to cover for the others weakness. For example, a troll and spellcaster working together are more dangerous than either alone, because the spellcaster can give the troll resistance to fire while the troll can tank for the spellcaster and use it's reach to provide battlefield control.
And it exists on the players' side of the screen, too!