I reverse engineered part of the CR, using the monster building guidelines in the DMG.
In the DMG, there is an Offensive CR, and a Defensive CR, which are supposed to be derived using certain instructions (some monsters fail to follow those instructions--with my personally bafflingly bad offender being the Coautl, who is only CR 4, while their Defensive CR is actually 13, and they should end up somewhere closer to CR 8).
Offensive CR: The minimum damage in the Offensive CR column for CR 1 plus is identical to the hit point total of a d8 character (max hit points at first level, as usual) with a +1 Constitution mod. So Offensive CR seems to represent a monster that could take down (assuming all attacks hit) what they considered an average PC in one round.
Defensive CR is harder to reverse engineer, although from levels 1-20 the hit point increase is exactly 15 per level. CR is supposed to assume the monsters nova with their best attacks on the first three rounds, and you average offense over those three rounds (and presumably defense over those three rounds where the monster has the ability to increase their defenses--but I don't know whether or not this is actually considered). So I put together what I expect is the assumed default party and looked at some numbers. For the default party I made a champion fighter, a thief rogue, a life cleric, and an evoker wizard. Since the designers are on record as saying that they assume a primary stat of 16 throughout the character's career, I did the same. Feats also would not be considered. I don't know which fighting style they gave the test champion, nor whether the rogue dual-wielded or reserved their bonus action for Cunning Action, so I played around a bit with both (I used a shortsword in either case). I limited spells to what could be found in the Basic Rules, because I doubt the designers went beyond that. I calculated average DPR with at-will attacks, DPR on a one-round nova, and average DPR on three-round novas. I determined what percentage of the monster's total hit points the party's DPR accounted for. I used average numbers rather than rounding down.
The numbers for Defensive CR were...inconclusive.
-As an average over the results of 20 levels, the 1-round nova damage averaged around 80% of the monsters hit point total, depending on party configuration. However it increased as the party level increased.
-A similar average for the the 3-round nova numbers (and this gets really tricky with spellcasters, because choices have to be made on which spells to cast) were coming in around the upper 60%s, but again in this case, the numbers were increasing with the party level, though not as much as with the 1-round version.
-For the at-will DPR, the 20 level average was 35% with the sword and board fighter and single shortsword rogue. The numbers were around 40% for the greatsword fighter and two-weapon fighting rogue. In the case of the higher damage group, the numbers decreased as the party leveled. With the lower damage group the numbers stayed fairly consisent--there didn't seem to be a strong trend towards an increase or decrease over the 20 levels. The average increase in damage per level for the same party comes up at about 4.75.
Defensive CR: The best I can determine is that the default test party does damage per round equal to about 35% of the total hit points of a monster of equivalent CR (the average for levels 1-20), assuming they only use their at-will attack resources.
So is the assumption of defensive CR that you take out a single monster in 3 rounds? Would the numbers hit exactly 33% if I rounded down each individual number before adding them together? I don't know.
In any event, these numbers are supposed to correlate with these descriptions:
"A monster's challenge rating tells you how great a threat the monster is. An appropriately equipped and well-rested party of four adventurers should be able to defeat a monster that has a challenge rating equal to its level without suffering any deaths. For example, a party of four 3rd-level characters should find a monster with a challenge rating of 3 to be a worthy challenge, but not a deadly one." - DM's Basic Rules.
Based on XP, a party of four facing a monster of CR equal to their level is a Medium challenge, and the low side of the Medium XP range at that. (Except for levels 5-7, when it is, believe it or not, an Easy challenge). A medium challenge is described as follows:
"A medium encounter usually has one or two scary moments for the players, but the characters should emerge victorious with no casualties. One or more of them might need to use healing resources." - DM's Basic Rules.
I did tweet Jeremy Crawford on January 4th asking about the actual derivation of the CR numbers, but haven't received a response. I'm apparently not good at phrasing the questions, because I tweet 2-4 times a year but rarely receive a response. Perhaps someone else would have better luck. Based on all the information they gave us in Xanathar's, I doubt they are intentionally keeping the math a secret.
