Does anyone know why the decision was made to design CR the way it is? Why is a CR X monster intended to be suitable for a party of level X adventurers?
It's only approximately that according to the latest guidance, IIRC. It is also, probably not coincidentally, how 3e, which introduced CR, did it as a default. (Though it did more than one monster in a somewhat simpler way - two monsters of CR-2 were 'equal' to a single CR monster. So if you wanted a party of 4 to fight an equal number of monsters, the baseline would be 4 monsters of CR -4. That is simpler than 5e's encounter budgets and multipliers, IMHO, though now that I type it, maybe not vastly so, especially if working out encounters for a party of 5, or wanting to outnumber a party by some less convenient ratio.)
If I had been designing CR, I would have built it so that one CR X monster was a match for a single level X character,
Actually, in 3e that is how it worked. If that hypothetical 5th party member turned on his 4 allies, that'd be a 'standard' encounter - standard encounters were meant to be speed-bumps that incremented party resources down by as much as 25%, not even-money fair fights. (I mean, how it was supposed to work, obviously the CoDzilla turning on you would be different from the fighter doing so.)
and a party of level X adventurers would be matched against a party of CR X monsters.
Well, that's how 4e did it, and 5e was in part motivated by a violent anti-4e reaction, so it couldn't really take anything straight from 4e if it could possibly be avoided. Another side of that coin is that 5e had a mandate for 'fast combat' without resorting to using minis or any sort of play surface, so reverting to a tactically simple or 'static' 3e style combat (defaulting to PCs dog-piling a lone monster) also made sense, from there the 3e default CR structure also had to have made a certain amount of sense.
It didn't exactly dovetail neatly with Bounded Accuracy, though, so a comparatively simple formula for multiple monsters couldn't be devised, and we got what we got. :shrug: It's also not terribly dependable, so, rather than lose sleep over it, I say just 'design' combats by feel, you will, if you don't already, develop a feel for what it takes to challenge a party of D&D characters - or even just go full-on "status-quo"/sand-box style and don't design encounter to be a certain level of challenge, at all, just design the setting and let the chips fall where they may, it's also a legit DMing style, and players will adapt to it after a sufficiency of TPKs.
Which was fine, but it relied on using solo and minion types to create larger or smaller encounters.
The simple numeric way that 4e did minions & solos (and elites) wouldn't have played with BA, but, ultimately, much-lower-CR monsters serve the same function as minions (if even less durable, since they can die to a failed save:1/2), it's just more complicated a substitution than a simple 4:1 ratio. And, 5e Legendary monsters are essentially designed to be Solo-worthy, even delicately lifting some tricks from later 4e and filing the serial numbers off to get there.
With BA, though, a normal monster can serve as a 'standard' or 'minion' or even a sort of elite or poor 'solo' depending on it's CR relative to the party, and a Legendary one can, similarly, play like an elite if lower level than the party, or Solo if of similar level. That was part of the point of BA, to let one monster stat block serve the DM across a number of levels.