I think you and
@EzekielRaiden are both somewhat correct here: Gygax knew what he was doing and to a decent extent tried to balance most parts of the game as best he could given the limited data he had to work with; and at the same time he was also quite intentionally unfair in some ways and - again quite intentionally - left some things unbalanced..
You're using two different senses of the word "balanced"--and only one of them is one I consider valid or appropriate.
The second, which I oppose, is "things being made precisely equal or distributed perfectly identically." This is an almost exclusively pejorative view of balance, as almost nobody actually wants this, almost everyone agrees that it is bad TTRPG game design, and to the best of my knowledge no actual games work by such a standard on anything but a very small scale.
The first, which I embrace and have explicitly said so many times, is "things being set in such a way so as to achieve a desired result." For Gygax, as I noted in my spoiler block, one of the key considerations was heist-focused play. The principle of a heist is that you are a comparatively weak invader, trying to sneak in and abscond with that which doesn't belong to you. It's pointedly
not war, because in war, you'd be marshalling forces approximately comparable to those you fight against. Instead, it's spec ops! You are a small, crack team of experts trying to leverage that expertise just right so you can get in, get the goods, and get out in one piece.
In order to achieve that end, it is
absolutely essential that the forces you oppose be overwhelmingly powerful...IF you play by their rules. Because that's how any bank-vault heist works. That's how any Great Train Robbery works. That's how any Mission: Impossible story works. We pay attention because it is awesome to watch a weak-but-clever figure overcome a far stronger opponent.
From this perspective, the thing you call "intentionally...unbalanced"
is not unbalanced. You are absolutely correct that it is not a perfect 1:1 matchup (indeed, it's pretty much the antithesis of a 1:1 matchup!), but equality of forces is
not what "balance" means in game design. What it is, is intentional design to evoke a particular mental and emotional experience: the fear of being discovered, the certainty that you cannot play by the defender's rules, the thrill of finding a clever way to defeat overwhelming opposition. And all of those things can be pretty cool! But they do, in fact, require that you be going up against opposition that you
cannot defeat on its own terms. Hence, balance within that context requires that the odds be, in a sense, "
only just surmountable," rather than "totally
insurmountable."
Other experiences require other approaches. This is why an effective, well-designed game does not provide loosey-goosey "ehh, eyeball it" rules that barely rise to the level of suggestions, let alone guidelines. Instead, it provides clear, concrete advice on the ways its tools may be turned to various ends, what you can do when testing its limits, and what
effect there will be from twisting stuff in a direction well outside designed parameters.
A well-designed D&D-like game, that recognizes the historical importance of the heist-centric model with the understanding that that model is not what most people actually
want out of a D&D-like game, does not tell you to make wildly unbalanced combats, as in, combats where you have absolutely no idea whether they'll achieve the desired experience or not. Instead, it tells you "this is the experience this tool was designed for; these are the factors that went into that choice; these are the common places that that can break down, and ways you can address them; these are additional tools you can deploy when seeking adjacent but distinct gameplay experiences."
5.0, instead of doing any of that, defaults to, "You could do X. Or you could
not do X! It's up to you to decide." Other than the CR-generation rules (which, based on reports I get from others, are not particularly useful), the few places where they actually deign to give any advice at all, it's worthlessly vague, e.g. the thing saying how to give XP for non-combat encounters
quite literally says to pretend that it IS a combat encounter and then award XP commensurate to that combat, without even the slightest hint of, y'know, HOW to translate a non-combat encounter into the rules of a combat one! It's
ludicrously bad. If you're already a skilled DM, you already know how to do this stuff far, far better than such a lame and worthless suggestion; and if you
aren't a skilled DM, that advice tells you nothing whatsoever.
Gygax presenting "only just surmountable" threats IS balanced,
in the context of a dangerous heist. Because you expect a dangerous heist to be extremely risky, and prone to failure if any of the steps of the plan go wrong. You expect that if you get caught by the guards, things can go south extremely quickly. You expect that there is little to no reward for doing anything except getting out with the goods in tow. Etc. The encounters are, in fact,
balanced, not because they put two forces in precise equilibrium (a ridiculous and strawmanning mischaracterization of balance), but because they
correctly make the heist opposition too dangerous for a frontal assault so that the players
must find another way.
