Haven't looked at their monsters very hard, but that definitely lowers my opinion of 5e.
5e monster design reminds me the most of 2e monster design with more predictable numbers and a level to help you judge what the power level of the monster is.
You've really changed the intent of that section. It's not worded nearly that strongly. Nothing says that your characters will die (because, as you know, they often won't). I mean, as far as I'm concerned, those pages of the DMG are total garbage and I never learned that part of the book very well, but even looking at now it's pretty clear that the writers at the time weren't very confident in it, and didn't want it to be taken too literally.
I do think that you are forcing your opinion on the text. I don't have the book in front of me, so I can't confirm exactly what you said, but there is a footnote at the bottom of the XP chart that talks about why the XP chart doesn't go below or above a certain level. I can't remember it's exact text, but the basic gist of it is "Encounters below a certain level don't use any resources from the PC and therefore aren't worth XP and encounters above a certain level are undefeatable and therefore no XP value was included. But that if they somehow managed to defeat an undefeatable monster, you should figure out the XP yourself."
It also includes the following recommendation on what monsters you should use in your game:
10% easy (EL lower than party level)
20% "easy if handled properly" (whatever that means...)
50% challenging (EL equals Average Party Level)
15% very difficult (EL 1-4 higher)
5% overwhelming (EL 5+ higher)
It explicitly calls anything EL 5 or higher than the average party level overwhelming(which is even harder than "very difficult").
And none of that stuff is on the SRD or part of open content either; I'd argue that anything that's not there isn't really part of the core of the game, especially given how many people (and companies) use the SRD as a primary resource.
Neither are Mind Flayers or Beholders.
The reason they AREN'T part of the SRD is the exact opposite reason. The XP chart for calculating XP was considered so intrinsic to playing and running D&D that WOTC felt if they left it out of the SRD, people would still buy the book and wouldn't be able to copy it precisely.
I mean, my D&D characters rarely look anything like what one would make from the original PHB. The game is about creating your own experience. The rules are suggestions; they're not inherent to the game like the rules of, say, basketball or chess. If you double dribble, or move a piece where it isn't allowed, that's breaking the rules. If I choose to play a game where Fireball does 2d6/level, I'm still playing D&D.
There's definitely some debate about this. It's commonly accepted that house ruling is fine in D&D, I agree. But it's also common to house rule some of the rules of Football when it's played out on the playground outside of schools as well. It's not real Football, but it's close enough and the rules are often changed to allow for things like the wrong number of players, a smaller field size, and difficulty to handle with no dedicated referee.
It's simply the culture that has risen around D&D that is perfectly accepting of changes to the rules. I'd argue, however, that by pure definition any game that changes the rules isn't "D&D" in it's purest form. In the same way that houseruled football isn't technically football. We allow these changes because they often make the game more fun for us. That's fine, people should make the game more fun for themselves.
However, I'm not going to say "No one really expected you to play with 11 players on the field at one time in football. That rule wasn't seriously meant to be followed. Look how well the game works when we play with only 5 per side. I think the creators of the football rules were just writing down things that weren't meant to be followed."
You may not understand the reason for the rule and your particular group might be able to run without it just fine. However, I don't doubt for a second that a LOT of thought was put into everything written in the book. Nothing would have been written and put in the book unless something thought it was important to be there and had a good reason for it.
I see nothing terribly wrong with contrived survival on occasion. It happens all the time in every venue of adventure fiction.
I hate using them. The difference between the two is that in a story you are reading it can often feel like "Oh, those guys were lucky and something odd happened to save them". In an RPG, where all the players know the rules, contrivances often come across as "Oh, good, the DM had to step in and save us because we were all going to die."
As a player it feels....unfulfilling. I'm this cool hero with all these cool powers and I should be able to defeat my enemies and triumph...or I should die if I fight monsters too powerful for me. If Elminster shows up and kills the monsters for me, I feel quite a bit less heroic. I also feel less needed. After all, if Elminster showed up to save us, why isn't he just finishing the rest of the adventure for us as well? He can do it better than we can.
Even if it's just "random rocks that fall from the roof" in order to kill the monsters for us, it still feels like we SHOULD have lost, but instead got away simply because the DM took pity on us. It doesn't feel very good to be pitied.
That said, if you really killed too many PCs, somehow I doubt it was the rules' fault; and you yourself posted above that you had an easier time understanding monster difficulty with 2e than with 3e. An unintentional TPK is likely the result of the DM or the players doing something they shouldn't have.
It was. I used a monster that was too powerful for the PCs. Though, I didn't know that until about the second or 3rd round of combat when it had easily finished off 2 of the PCs and was barely hurt. It was at that point that I thought "Oh, crap, I guess it's too hard for the PCs to hit this monster and it does too much damage. They are all going to die. If I knew that in advance, I never would have used this monster."
Other times, players have gotten a character killed when they clearly shouldn't have attacked in the first place
What make this "clear"? I understand that after playing D&D for a while, you start to get an innate sense of the power level of monsters. I learned over time and I stopped using overpowered encounters when I got better at judging the power level of monsters.
The same thing works as a player. If I've never played D&D before or I've played only a little bit, how am I to know how difficult a Beholder is(or even WHAT a beholder is)? If I run into battle with it at 3rd level because I figure I must be able to beat it because I've defeated every other monster up until this point, does that mean that I'm an idiot?
Maybe the DM should provide input at this point...but what tells the DM who has never used a beholder before that it can't be taken out by 3rd level characters?
In any case, none of those scenarios would be ameliorated by anything like a CR/EL system, and the responsibility for the outcome was always with the people at the table.
I'm not sure about that. Was it the DMs intention to put the players in a situation where they had to run away or use defensive tactics? Did he sit down and say "This battle is going to go poorly for them. They won't be able to win and will have to run away"? Or did the DM plan that the monsters would be defeated easily and then the PCs would proceed down the corridor to fight the next encounter? If the DM planned on the monsters being defeated easily and wrote up a bunch more encounters that take place later in the dungeon then the DM has failed.
It's the DMs responsibility, yes. Is it his fault? Not if the game didn't give him the proper tools to predict this in advance.
Tactics are fine and dandy, but if you have +3 to hit and 20 hitpoints and come across a monster who has an AC of 30 and an AOE attack that does 30 damage, I can tell you that no tactics you come up with are going to stop every member of your party from dying. This is obviously an extreme example and a basic understanding of the rules SHOULD stop this, assuming the DM took the time and effort to read through the stat block in advance before rolling for initiative(which isn't a guarantee if you're low on time). However, a more reasonable example where a monster does only 10 damage and has an AC of 19 can seem like a good idea at the time. Until you run the monster and realize that 2 rounds later, the PCs are all dead because 2 10 damage AOEs kills them all.
And to top it off, failure is part of the game; just because a character dies and no one meant it to happen doesn't mean the game is broken. What kind of a game has no losers?
Well, most Epic fantasy books and movies have...either no or VERY few losers. Only Boromir dies amongst the entirety of the Fellowship(unless we count Gandalf, which I don't).
The hobbits weren't attacked by all 9 nazgul in a cave that made it impossible to retreat at the beginning of the story. Because that would have ended the story in failure and wouldn't have been interesting to read/watch.
The DMs job is very similar to that of an author. Make the story interesting and don't kill off all the PCs. However, it's made even harder because you don't have absolute authority like an author. You have to rely on the dice, stats, and rules to decide part of the game. When those dice, stats, and rules say "Everyone dies", it can ruin a game.
I like when the rules give me tools to help me decide WHEN I'm about to kill everyone so I can make that decision for myself instead of doing it accidentally.