That's the fault in your argument. 4e did two things: it made the game easier to run and it "disrespected the game's history". Those two things are not inextricably bound together.
They were - well, the former was a sub-set of the latter. D&D was traditionally tricky to DM, DMing was more of an art than a science, say. Changing that was one of many attempts to make the game better that ended up sending some fans into paroxysms of nerdrage for, as I put it "disrespecting the game's history." (I think that's a reasonably nice way to put it, sorry if it bothers anyone.) The DM-player dynamic in D&D prior to 4e was simply different, and opening that dynamic to change - among other things - triggered some very negative reactions.
5e encourages a DM-player dynamic more like that of the classic game.
A fair bit of biography here masquerading as universal.
Eg my 4e game - with the Rod of Seven Parts, the Eye of Vecna, the Sword of Kas, and an Orcus-slaying party that includes a dwarven cleric/fighter swapping between hammer and axe and an elven ranger-cleric with a demon-slaying bow - does not disrespect the game's history. That list of stuff is basically the roll call for D&D nostalgia!
Sure, it didn't for you & your group. I also found plenty of nostalgic D&Disms in both 3e and 4e - my second 4e character was a reprise of the positively cliche 'high-elf fighter/magic-user with a wand of fireballs' archetype from among my earliest D&D experiences. Oh re-imagined, more sophisticated and lampshading a few things, but still a lot of nostalgic fun. But, for others, the change in basic mechanics, class balance, or even very basic labels, like spell levels being simplified to match the class level at which they were gained, were positively intolerable. You can argue that the fault for that reaction lay with them, and I'd agree, but the edition war happened, and 5e had to react to it.
So it's, y'know, 'reactionary.' Ironic, perhaps, but it's working.
Nor does my game, or other 4e games I've read about on these boards, involve "cookie-cutter" or "totally predictable" encounters.
Nod. The basic, dead-easy, totally intuitive, same-number-of-same-level-standard-monsters encounter would have been a 'cookie cutter,' if it were the only thing. It's wasn't. Sufficient lack of familiarity with the system could leave one with the impression it was, though.
(I can't address how they compare to 4E because I never played that beyond Keep on the Shadowfell, which I'm told is not representative.)
I ran & played a lot of 4e. It might as well have been a 'different game' (as it's critics often ranted) - there was so much going on in terms of player options at charge/level-up and in play, and there was so little to DMing it the DM could even start to feel like just another player. I've known DMs to run 5e in a way that resembled 4e, but they did it by essentially running 4e - using 4e rules for movement, actions, &c.
With DM Empowerment, you can get away with that.
That's kind of a problem for the game, then. The designers of the game identified some time ago that the biggest factor limiting the growth of the game is the rate at which new DMs can be recruited and 'trained'; at some point, the number of players at the table gets to be too large to handle effectively, so someone needs to step up and become a new DM to allow the table to split into smaller, more manageable parties. But if the perception is that DMing is hard and takes a long time and lots of effort to get 'good' at it, then people aren't going to be all that willing to step up to DM. (And anybody who has tried recruiting new DMs for Adventurers League or Living Forgotten Realms can sympathize with this.)
But, at the same time, you can't afford to lose existing DMs - you especially can't afford to turn them against you.
Thus, the impetus for encounter-building guidelines, with the implicit promise that, if you crunch the numbers correctly, you'll come up with a challenging, not overwhelming encounter that your players will enjoy.
...Of all the versions of D&D that had encounter-building guidelines, 4th edition's was the most useful
And it did lead to remarkably rapid transitions from new player to new DM. I saw it myself many time during the Encounters period, players quickly picking up the game, then moving quickly to picking up an adventure and running it.
But I was also very often the oldest guy at the FLGS. When I wasn't, like as not, my fellow grognard would be there recruiting for Pathfinder.
4e might be the pinnacle of balance, but balance brings predictability. And IMO, predictability breeds boredom.
Decent encounter guideling bring greater predictability - for the DM designing the encounter. Though, really, it's more a matter of reduced unpredictability - the only thing a DM, like the allegorical cat-herder, can really predict is that the players won't do what he predicts.
That doesn't mean the encounters he comes up with can't be quite varied (and thus seem unpredictable to his players), just that he'll have a better idea of when he's created a probable rollover or a risk of TPK.
D&D didn't have rules for balanced encounters for decades, and it was the most popular time for the game
That's deeply faulty logic. Yes, D&D was fad in the early 80s when it was badly balanced & hard to DM, it didn't become any less imbalanced in the 90s, but lost virtually all of it's popularity anyway. Broken = popular isn't even a valid correlation.
What you do need, is a DM who is familiar with both the rules and their players if you want to create encounters that your gaming group will enjoy.
Absolutely. And more true the more encounter design tends towards art over science.
What I think would be great is an explanation on how to adjust the encounter guidelines based on different expectations of encounters per day. That I think would be interesting.
It would be interesting. But it might encourage problematic play styles (the infamous 5MWD, which has always been a problem with D&D, and which 5e, expecting 6-8 encounters and having a more complex than ever variety of resource mixes, is particularly vulnerable to).
In my experience, 5E fights run exactly like 3E/PF ones if you crank up the difficulty and put a bunch of terrain stuff in.
I can't say that matches my experience, I don't see 3e's level of 'rocket tag' in 5e, and being outnumbered seems to be more problematic in 5e than it was in 3e.
5e, at very low level, does feel like classic AD&D, to me. After that it's prettymuch up to the DM to try to mod it into what he's looking for.
So my advice to anyone looking for help in creating encounters is to a) get really familiar with the mechanics and monsters, and b) get really familiar with how your players typically play, and then create encounters that feel right.
I would add: don't hesitate to adjust them on the fly.
I take issue with the bold part there. The assumption is that one is expected to improve play ("get better") in some manner that makes battles easier or less dangerous for the players. That is one way / style of play.
It's a classic style - 'skilled play' - going all the way back. Yes, the idea is that not only do characters get better by leveling, but players get better through experience. So experienced players re-starting with 1st level characters will be more effective than they were their first time 'round and 1st level. For most D&Ders, that cycle has happened many times, and there's only so many differences between one edition and the next for them to un-learn & re-learn before they're at the top of their game again.
5e encounter guidelines seem not to assume a lot of player skill (or closely related but distinct 'system mastery') but provide plenty of room for both to be developed and rewarded. So it gets 'too easy' and the DM has to dial it up. Not a new issue. By the time that happens, the DM should also have enough DM skill & system mastery to deliver.