First off, I am genuinely curious and trying to be helpful but if I am bothering you please tell me and I will stop.
Secondly you kind of remind me of where I was regarding encounters during 3.5
No worries. I appreciate all the advice here. I clearly still have a lot to learn.
Well, first off how challenging do you think it ought to be? I do not run 8 encounters per day either. The players always regard an encounter as deadlier than it is, in my experience. By challenging, I do not mean the DMG categories, I mean in plain language what is average, what is the top end of challenging.
On my part, some encounters are easy to make the party feel powerful and some are hard to make them feel they have earned their win.
I can provide only a vague answer: I want the players (and their characters) to have the opportunity to make interesting decisions. Whether it's in a social encounter, exploration in a dungeon, or a combat, I want their actions (and the resolution of them) to be impactful.
Does the cleric heal the warlock who has taken substantial damage or use her action to embolden the barbarian's might to further threaten the umber hulk?
Does the rogue betray her position to the troglodyte warcaster with a surprise attack and disrupt the spell, or does she stay in the shadows hoping for a better opportunity?
Does the monk leap over the chasm to stop the sacrifice of the baron's son, knowing that he'll be the target of the ogre bodyguard's push into the chasm?
Does the sorcerer drop his last fireball centered on himself in desperation, surrounded by duergar reavers, in the hopes he can take enough of them down with him?
These moments simply don't come up often enough in my 5e combat situations. As designed, the monsters don't tax the party's resources enough that characters have to give up opportunities or make meaningful decisions.
I generally ignore magic items, I mostly pay attention to action economy and damage output.
The last time I did that, it was to the detriment of the game. (Read my previous posts regarding the Wand of Reverse Gravity.)
What is the average damage per round from the party, multiply by the number of rounds you want the monster to last and give it that many hit points and adjust the action economy to make it a challenge to the party. The actions of the bad guys has to match or even over match the party in the early rounds and be somewhere near while they are up.
A dragon has 3 attacks and 3 legendary actions against a party or 4 with may be 5 actions per round at low water to high as 8 not including bonus actions (depending on composition). So a party of 8 will have at a minimum have 8 actions, probably average at 12 or so. That dragon will be near dead in a round and they will probably burn out its legendary resistances in a round also.
This is my suggestion, though to be honest I mostly eyeball it.
The most successful 5e combat system I had was completely abandoning the encounter system and monster design. I averaged the stats of every character (HP, damage bonus, average damage, AC, etc.) and completely retooled every monster they encountered. Did I want a monster to drop a character in 3 hits? Then its damage was 1/3 the average character HP.
But the issue is, we were playing a professionally produced adventure that I bought. The party composition was standard, and no one had a character with ridiculous stats or magic items. I shouldn't have to do that level of customization. The system should basically work out of the box.
They did, it was called fourth edition and we remember what happened.
Yeah. I did tinker with 4E as well. (Mostly with the pretty common houserule of lowering monster HP and increasing damage - just to speed up combats.) Even by mid-levels (paragon tier?), it wasn't unusual to have 2+ hour combats. There was movement, buffs, and a handful of tactical options that could be checked off on your Character Builder sheets (Daily, Encounter, etc.), but there was a boring sameness to most combats.
I think one of my biggest gripes is that D&D has too many combat encounters for many of them to feel worth doing. And the ones we have also take too long. I want high stakes, excitement. I want combats that come down to the climactic 2-3 rounds that actually matter.