So, when this happens, do you note what the difference was?
Perhaps in the former case you couldn't roll above a 5? Perhaps in the latter case you rolled half a dozen crits in the same encounter?
Maybe I'm confusing you with a different poster, but weren't you the one who just claimed to have 38 years of experience? No offense, but this post reads like weaponized incompetence on your part. Like in those 38 years you couldn't figure out how to estimate the difficulty of an encounter. Frankly, I find that a little hard to believe.
Yes, RNG is always a factor. I prefer it that way. I would think, given your stated preferences, that you would prefer it that way too. However, as DMs with some experience, it's not that difficult to get a feel for how challenging an encounter will be on average.
If the elder dragon rolls nothing but 1s and the players roll nothing but 20s then they may steamroll it. And if the players roll nothing but 1s and the lone kobold rolls nothing but 20s it may steamroll them. (Obviously abilities that are save for half change this a bit, but let's leave those aside for the sake of simplicity.) However, we can pretty safely assume that for a 4th level party the elder dragon will most likely result in a TPK if engaged directly, whereas the kobold will probably pose a trivial encounter.
After we've played a session or three, we can probably start to get a feel as to how the party is performing relative to the encounter building guidelines. If they're over performing we can increase the challenge a bit (this doesn't even have to be additional creatures; it might simply be something like terrain that favors the monsters). If they're under performing, we can reduce the challenge.
Maybe I'm off base and the guys I game with and I are gaming geniuses who managed to crack a code that no one else has managed. I strongly doubt it though. I'm pretty sure that any DM with a modicum of experience under their belt can manage the same. For those DMs who aren't experienced enough, setting the guidelines on the easier end of the scale is a good idea, because they can gradually learn to get a feel for the system without accidentally slaughtering their parties.