Because smaller numbers are easier to deal with. The smallest dice that gives the desired granularity should be used. To me it's definitely easier to check wandering monsters with 1 on a d6 rather than 18-20 on a d20 like in 5e. I can't think of any reason to use the same dice for everything. It seems to be an aesthetic thing.
But there are HUGE systematic advantages. Just for example its possible to apply ability score bonus/penalty to anything that is rolled on d20 when its appropriate. Wandering monster checks might not be one of those things, but surely surprise checks (IE use of stealth) most certainly is! I wouldn't recommend something silly like rolling d1000, but there's no difference in my mind between picking up a d6 and rolling it and picking up a d20 and rolling it.
If your character somehow secured a level in the gameworld I would allow them to use it. I just don't mind that the system doesn't provide a built-in way for humans to do that.
Right, but that system is detached from ability scores, the prime determinant of what you are good at, and has only (in AD&D anyway) a resolution of 25% increments, which is a bit crude as its pretty difficult to model "any intelligent person trying to dope this out" which would be maybe a LOW chance of success, but its POSSIBLE (Imagine yourself in a mine with some equipment, you could probably determine if a passage sloped, maybe not with great reliability, but you'd have a CHANCE of it).
In AD&D this extended on to many other things, like "no, you're a wizard, a cosmic vortex will open up if you swing a sword!" It was just not an acceptable approach to RPG rules to a lot of us, never has been.
IME games that try to do everything don't do anything well. I like games that have some focus. This is just a rule of thumb on my part though. It's definitely not a priori true to me that a game should be able to handle anything a character could try to do. Limitations can produce better, tighter gameplay. I'm the sort of person who doesn't necessarily mind if you can't jump around in an FPS videogame, as you often can't anymore (to look at another example). D&D is relatively quite a very free, open-ended game regardless of whether there are a few class-specific abilities or not.
Nobody is claiming that a game should 'do everything'. Just because it has a more generalized system where you can actually adjudicate most anything someone will try in 4e doesn't mean the game is 'good for everything' or 'does everything'. It simply lacks the impediments to doing things that COMMONLY COME UP IN PLAY in all editions of D&D.
As for hard limits built into the rules, me. I don't agree that they make a better game. I don't see that 3e or 4e are 'worse' games in any measurable sense than AD&D or other 'classic' D&D, yet they contain few of those traditional hard limitations.
I suspect it's often used improperly.
We'll just have to agree to differ in our opinions on that. I certainly have no intention of playing games where the philosophy of design is to not give me tools I can use because its POSSIBLE someone else might not use them "correctly" according to the ideas of the designer. I don't think game designers should feel obliged to include things they won't use or aren't interested in when they design games, or things that are directly opposed to the tone/genre/agenda of the game, but D&D was always a game where you could do a lot of things. As a generic FRPG aimed at the bulk of the FRPG audience's needs it really should try to be inclusive.
I agree that would be undesirable, but that never comes up in my 1e or classic games. That seems to be more of a theoretical concern. I mean in the example with climbing across the wall, I don't think if I were "really there" that I would want to try climbing it. The result was absurd and not realistic at all. I think when the resolution system is too easy and predictable and samey (ability check for everything) you just do the first thing you think of, which can feel sort of fun and freeing sometimes, but to me is not ideal for "serious" gaming.
If I were to run 3.x or Pathfinder I would actually try to use all of the rules and modifiers for the skills.
Again, we'll just have to agree to disagree. There were tons of times I can recall in 1e and 2e where I thought "Yeah, I could petition the DM to come up with some sort of decision about how hard it would be for me to climb that thing, but why bother? We'll just find some other way even though climbing is perfectly logical because who knows what can of worms this will open up with the DM?"
I don't know about the specific situation in your example, you say it was absurd. OK, but that's not necessarily a fault of the system, it sounds more like to me that the DM made the skill check too hard. Many of the older skill systems were quite crude, including that in RQ that you were talking about. There was a fixed chance of doing ANYTHING, with some very awkward and not well explained 'difficulty multiplier' that you might or might not apply to some things. 4e, as an example, has a MUCH more sophisticated and usable skill system where the character's level, training, relevant ability score, and possibly other factors all count in, and are measured against a determined DC that reflects the difficulty of THAT SITUATION specifically. It works quite well. Furthermore there is a pretty large distinction between characters, so you don't just get people doing any old thing that came to mind. They do what they're GOOD at. Usually they do things where they have relevant powers/feats/items to give them help too. This is another area where the 4e generalized approach shines, you can easily apply things like enchantment bonuses from items, which was pretty much impossible to do in AD&D unless it was a weapon and combat, or else there certainly had to be a special rule about it (which again ran into the problem with the variable granularity of different dice).
I can appreciate the aesthetic of rolling different dice for different things, but I totally disagree that it has ANY mechanical advantages or any mechanical reason to recommend it. Its a purely aesthetic preference which has objective mechanical costs. Whether you care to accept the one in trade for the other is obviously up to you. However, I think it pays to be objective about the reasons for doing things.
My guess is that in the days when D&D was being first developed there was no reason to consider a d20 special. The combat system was chainmail and the 'alternate' d20 based system was an addition. There wasn't any real reason at the time to think of one particular resolution system as better than another, and various elements were grafted on from various sources and brought their own dice conventions along for the ride. Clearly the designers of games post-D&D however saw the advantages of unified dice mechanics quite quickly. Virtually every game designed SINCE D&D has had them, and D&D itself has slowly but steadily moved in that direction. I see 4e being the apex of that movement and as a superior design because of it. Its really a pretty refined system that has huge advantages.