So the 5e Bandit Captain is roughly as dangerous as a level 5 PC fighter.
HP are inflated on monsters because it is easier than adding self healing abilities.
Look at a level 5 Fighter, first without a subclass. We'll make them a TWF as well.
Dex will be 18, two weapon fighting style, pair of scimitars for 3d6+12 (23) damage, plus an action surge for +15 per encounter damage, all at +7 to hit. They have studded leather, so 16 AC.
10+6*4 + 2*5 = 44 HP, but also has a second wind for 1d10+5(11), so 55 HP of soak.
The fighter also has a subclass. If we use a middle of the road one like Samurai, this fighter has +15 temporary HP/day, for a total of 70 HP of soak in a single fight (advantage on attacks and losing an off hand attack are a wash)
This is a far from optimal 5th level fighter.
In comparison, the Bandit Captain deals 2d6+1d4+9 (18.5) DPR at +5 to hit and has 15 AC and has 65 HP.
It is basically inferior in almost every way, but is in the right ballpark at least.
The same holds of most CR 2-3 fighter-type humanoid monsters in 5e; they are all in the ballpark of a level 5 PC, some optimized some under optimized.
The fact that they often end up with inflated number of HD can be treated as a simplicity of emulation. Drop their HP by 30ish. Add in a second wind ability (reaction) that heals them for 15, grants them 15 temporary HP, triggered by being reduced to half HP or less.
Now that Bandit Captain has 5d8+10 (32 HP) and a 30 HP heal, for a total of ... 62 HP.
Of course, this is sort of pointless, because a player is unlikely to see the difference. On a PC, such an ability would be a mechanical knob they'd have to tweak. It would refresh every long rest. Maybe you'd add in an extra feature, like it grants advantage on attacks for a turn.
On a monster, the DM has enough stuff to do. So they give you some stats that emulate a level 4.5 PC fighter reasonably well with as many mechanical knobs filed off.
Something similar happens with berserkers. They have rage damage resistance baked into their HP! It is a lot easier to double their HP than to give them resist all.
Now, this does mean they aren't specifically vulnerable to psychic damage, unlike a bearbarian. They must have a different totem. (berzerkers are a weak CR 2, closer to a level 3 PC than a level 4 or 5 one).
