Nothing's preventing a better DC system in conjunction with skill point though. And never EVER gating something behind skill ranks.
I am genuinely not convinced that a different skill point system would actually fix the problem. It's having to navigate between the Scylla of 3e's punishing system and the Charybdis of "put points wherever you want, because it doesn't matter."
I have no personal knowledge of 4e, so I can't really agree or disagree with you on if it's true of 4e. I did hear more than once in 4e discussions that DCs were supposed to scale with the group, and we all know that since I heard it on the internet, it must be true!!!
I have two thoughts on this.
First, I think that you are partially correct when you say that it's not about treating the rules as suggestions with no validity. The "rules" are valid. However, it's also clear that they are also just suggested rules. I went through the DM last year I think it was and listed a huge number of times it calls the rules guidelines(suggestions). I'll list just a couple now.
"Chapter 3, "Creating Adventures," provides guidelines for designing combat encounters using experience points."
"AS THE DUNGEON MASTER, YOU AREN'T LIMITED by the rules in the Player 's Handbook, the guidelines in this book, or the selection of monsters in the Monster Manual. You can let your imagination run wild."
And of course at least a half dozen times spread throughout the DMG where it says that the rules serve the DM, not the other way around. All of that indicates to me that the rules are suggestions for the DM to build on or override when he feels it should happen, not just when some serious issue comes up. They are encouraging house rules.
Second, 5e seems to be deliberately designed to force rulings. There are far too many "rules" or "guidelines" that are written vaguely and/or with common situations involving that rule that are just plain missing.
The designers are forcing rulings over rules in this edition.
As for your 5e book quotes, yeah, I know that's what it says, and I'm not happy about it. The deliberate vaguess is worse, because that means actively trying to design a game that isn't functional by itself.
Sure, I get that desire, and that’s probably why scaling proficiency bonus gradually worked its way back into 5e. But if you do have scaling bonuses, you either need to accept that things will get easier as you level up, or to have commensurately scaling challenges. And in the latter case, either the challenge must be tied to the fiction (so eventually the doors you deal with as challenges will be adamantine) or not (so despite the numbers having increased, everything remains exactly as challenging as it ever was).
Again, the way to dodge the horns of this dilemma is to actually include variety. Send the players to a location where some of the doors are just wood and others are adamantine, and give them that chance. Or, say, have the Paladin get captured by opponents who vastly underestimate her. Etc. Judicious use of weak opposition lets the rising tide actually be noticeable.
This might seem off-topic, but I'm wondering if it gets at the root of some of the disagreement:
When you "prepare" spells for the day, is your character aware of the # of slots, levels, etc.? That is, do the game rules map 1:1 to the principles of magic as understood by the characters?
My answer is no. But if one's answer is yes, then that would lead to different conclusions about stat blocks (and many other things).
In general, I assume the answer is yes, but would be willing to hear out an explanation from the DM why that isn't the case.
I probably wouldn’t do this myself, as I prefer using human judgment over a hard rule.
I mean, the rule can be very simple. If there's no interesting consequences for failure, just tell the player it happens (perhaps with gusto, if they heavily outclass the obstacle.) If there's no benefit to success, give them the opportunity to back out. (My players quickly learned to listen when I say, "Did you really say/do that?" or "Are you sure?" The former is mostly for separating OOC from IC, while the latter is my ritual phrase for such moments. I don't say it very often, because 99% of the time my players are cautious to a fault rather than too aggressive, but I know my players notice. Sometimes they even say yes, they really are sure!)
If so many people got the wrong idea about 4e, even those who ran it for a reasonable amount of time, how clear could its design ethos have really been?
Well, it would have helped if they both (a) actually read it (many did not, as their blatantly false claims demonstrate, e.g. claims that healing surges allow infinite healing), and (b) actually used quotes from the text rather than aggressively misinterpreted paraphrasing and summarizing. E.g. people still to this day claim that every combat encounter in 4e is required to be keyed to the party's level exactly. This is not only false, but trivially easy to disprove simply by reading the relevant sections of the 4e DMG. There is no ambiguity here; the book explicitly says not to do that, and instead to give a wide variety of combats (in encounter level, number and types of opponents, and terrain on which the encounters occur) so that the players have a rich and varied experience. This isn't hidden. It isn't buried in a mountain of text. It's right there, on the surface.
This is why so many fans of 4e get so annoyed with the pushback 4e still gets. A ton of it is not merely wrong, but
trivially wrong. Wrong in ways that anyone who had genuinely read the books should not be capable of.
Sad thing: I have not find a system, with a better balance of out of combat and in combat stuff.
Probably it is familarity, but we just like how the game is structured. We just don't need more out of combat rules. Maybe a bit more steuctured out of combat rules, because the DMG is sadly a bit unorganized in that regard, and improved rules are found in xanathar's guide and so on.
I find Dungeon World is quite balanced in this way. But it is also a much looser system than D&D ever has been. If you like rule structures for non-combat encounters, you might consider adapting 4e's Skill Challenge rules. The "Obsidian" houserule variant is a great place to start. SCs got a crazy bad rap in 4e but were actually a really good idea marred by (as was so often the case in 4e) EXTREMELY, PAINFULLY BAD examples given in the early WotC adventures.
Ok. But the players probably don't know what combat spells the NPC caster has either. So are you fine with adding new spells to the NPC in mid combat?
I am not, unless it is clear that they should have such things but it hasn't been specified what yet. If there was no way for the players to know prior to it being added, it's no different from the vast majority of other DMing where the DM improvises the content as they go.