Neither Xanthar or DMG system well designed for just 1 PC, so the fact they disagree or any claims to accuracy is going to be questionable.
I'm not sure if you can make such a system, because PCs vary so much in capabilities; parties make up for this with variation.
Like, a swarm of weak foes could be one-shot by a high-initiative wizard; or be a horrible slog for a solo rogue.
---
If you want to be serious, start with calculating the 3 round average DPR of each PC against (say) 16 AC (if they use spells, assume ~40% save rate and guesstimate how many targets you get. Secondary targets are worth 1/2 the damage value as primary targets are; so a spell that does 20 damage (0 on a save), and hits 2 targets, is worth 20 * 1.5 * 0.6 = 18 damage).
Multiply their HP by (AC-6)/10 for tankage factor.
Do the same for monsters.
The combat should last (monster HP times AC modifier) / (player DPR).
On average, half of the monsters will be alive on any one round. So take that many rounds, divide by 2, and multiply by total monster DPR.
How does that compare to PC HP? If way less, easy fight, of more than PC HP, crazy fight.
Now, AOEs will actually drop the monsters faster than this, but more of them will be alive on average; with the "half secondary targets" trick above, things work out. Second, you should model the PCs with stupid tactics; ideally, PCs should figure out who is most threatening and drop them first, or do something else clever. Leaving room for PCs to be clever with a fight that is otherwise hard when played dumb can feel rewarding.
---
Maw Demon. 0.7 AC multiplier, 33 HP for 23 effective HP
11 damage per hit, +3 to hit, +9 per crit. On AC 16 is 4.85 DPR -- call it 5.
Cackler. 0.9 ACx 10 HP for 9 effective HP.
6 damage per hit at +5 for about 3 DPR.
Last laugh for 0.5ish.
Rutterkine 0.6 ACx 37 HP for 22 effective HP
12 damage at +5 to hit for about 6 DPR.
poisoned and frightened is a bit nasty. Have to fudge that.
So your total effective HP is 104 and DPR is 28.
A level 9 duelist BM fighter with 14 con, a longsword +1, 22 AC and 14 con and 20 strength. 91 HP after second wind, times 1.6 for 146 effective HP.
+10 to hit for 1d8+8 (12.5) damage on 16 AC is about 10 DPR.
Can drop another 5d8 BM dice for 22.5 damage. Note that these are not calculated quite right, as they always hit; enemies have low AC, so we are over valuing them.
104 HP - 22.5 is 81.5, divided by 10 is about 8 rounds to drop the horde.
8 / 2 * 28 is 112 HP done to the fighter.
The fighter has a good chance of winning, even played poorly. It will be a long slog.
A PAM/GWM fighter would have a much shorter fight, but have lower effective AC.
---
A level 9 paladin with 20 attack stat and 22 AC has 34d8 of smite dice on fiends and 45 lay on hands.
At 12 con they have 67 HP + 45 for 112 HP (costs a round to LoH), at 22 AC that is 179 effective HP.
The smite budget is like 150+ damage. At 2 swings/round over 4 rounds (including a round to LoH) you'll get 4-5+ hits and drop everyone.
in 4 rounds, the enemy does about 56 HP. The paladin wins easily. Might not even need LoH.
I'm not sure if you can make such a system, because PCs vary so much in capabilities; parties make up for this with variation.
Like, a swarm of weak foes could be one-shot by a high-initiative wizard; or be a horrible slog for a solo rogue.
---
If you want to be serious, start with calculating the 3 round average DPR of each PC against (say) 16 AC (if they use spells, assume ~40% save rate and guesstimate how many targets you get. Secondary targets are worth 1/2 the damage value as primary targets are; so a spell that does 20 damage (0 on a save), and hits 2 targets, is worth 20 * 1.5 * 0.6 = 18 damage).
Multiply their HP by (AC-6)/10 for tankage factor.
Do the same for monsters.
The combat should last (monster HP times AC modifier) / (player DPR).
On average, half of the monsters will be alive on any one round. So take that many rounds, divide by 2, and multiply by total monster DPR.
How does that compare to PC HP? If way less, easy fight, of more than PC HP, crazy fight.
Now, AOEs will actually drop the monsters faster than this, but more of them will be alive on average; with the "half secondary targets" trick above, things work out. Second, you should model the PCs with stupid tactics; ideally, PCs should figure out who is most threatening and drop them first, or do something else clever. Leaving room for PCs to be clever with a fight that is otherwise hard when played dumb can feel rewarding.
---
Maw Demon. 0.7 AC multiplier, 33 HP for 23 effective HP
11 damage per hit, +3 to hit, +9 per crit. On AC 16 is 4.85 DPR -- call it 5.
Cackler. 0.9 ACx 10 HP for 9 effective HP.
6 damage per hit at +5 for about 3 DPR.
Last laugh for 0.5ish.
Rutterkine 0.6 ACx 37 HP for 22 effective HP
12 damage at +5 to hit for about 6 DPR.
poisoned and frightened is a bit nasty. Have to fudge that.
So your total effective HP is 104 and DPR is 28.
A level 9 duelist BM fighter with 14 con, a longsword +1, 22 AC and 14 con and 20 strength. 91 HP after second wind, times 1.6 for 146 effective HP.
+10 to hit for 1d8+8 (12.5) damage on 16 AC is about 10 DPR.
Can drop another 5d8 BM dice for 22.5 damage. Note that these are not calculated quite right, as they always hit; enemies have low AC, so we are over valuing them.
104 HP - 22.5 is 81.5, divided by 10 is about 8 rounds to drop the horde.
8 / 2 * 28 is 112 HP done to the fighter.
The fighter has a good chance of winning, even played poorly. It will be a long slog.
A PAM/GWM fighter would have a much shorter fight, but have lower effective AC.
---
A level 9 paladin with 20 attack stat and 22 AC has 34d8 of smite dice on fiends and 45 lay on hands.
At 12 con they have 67 HP + 45 for 112 HP (costs a round to LoH), at 22 AC that is 179 effective HP.
The smite budget is like 150+ damage. At 2 swings/round over 4 rounds (including a round to LoH) you'll get 4-5+ hits and drop everyone.
in 4 rounds, the enemy does about 56 HP. The paladin wins easily. Might not even need LoH.
