I'm A Banana
Potassium-Rich
Well how how about protecting the wizard niche? A rogue at high levels who has poured ranks into use magic device can very effectively compete with a wizard in the spell category. Not only can he use arcane spells but he can also use divine spells.
The wizard's niche should be protected, too. But "being able to use magic" is an overly broad niche for a fantasy roleplaying game, IMO. 4e's concept of combat roles is a great start, and I believe that can be expanded to cover noncombat areas as well.
I've always liked the Wizard archetype as a sort of "Magical Scientist." Research, knowledge, arcane lore, the workings of thaumatrons and thaumiolis, the structure of the planes that they then rip a new one...
That's a pretty narrow archetype. A magical scientist isn't a charmer or a trap disarmer, they're not a regular combat powerhouse, and while they should be able to participate in every challenge (they do need some combat ability, and some exploration, and some social graces), they really shine when getting the party Information. They know stuff, and what they don't know, they know how to find out. A Wizard is your fantasy Wolfram Alpha and Wikipedia all in one.
That means that if you want to be some sort of elemental kaboom artist, or some charming enchanter, or a crafty illusionist, or a dark-loving shadow mage, you're picking a different class (or at least a different build). You don't get to do all of that as a Wizard.
The rogue also with his back stab can compete with the fighter when it comes to doing damage in combat.
Then there is the cleric with his buff spells and access to heavy armor he can more than compete with the fighter to be the frontline tank.
It is almost impossible to totally protect niches there will always be some overlap.
This is why I am a fan of 4e making roles so central, though. It's possible to protect a niche if the niche is clearly defined in terms of what that character brings to the table. A little overlap is fine -- my pyromancer wizard is not too shabby about damage, but he's not like the party Strikers, who, in turn, can't affect as many targets as I could. And none of us are capable of the distraction power of the Paladin or the healing power of the Warlord.
Expanding on that concept, we have characters who get A+'s in various noncombat things: Intel, Exploration, Interaction, Recovery. Some characters are good at advancing the party through hostile terrain (think: a rogue who can find and disable traps and scout ahead for monsters). Some characters are good at dealing with NPC's (think: a charismatic paladin who can help anyone to feel like they can contribute). Some characters are great at learning about the dungeon or the threats, and no plot thread can hide from them for long (think: a wizard with scry and access to a library). Some characters shine when others screw up (think: a cleric who can heal the rogue that just failed to disarm the trap).
Those narrative roles, IMO, need to be as protected as the combat roles.
As for the spells you mentioned like spider climb it is should be in the wizard repertoire. Wizards are not as strong as fighters and they don't have climb as a skill so they need to be able to climb that cliff face. They can also use it to help other classes.
Climbing the cliff face falls into "exploration," so my theoretical wizard archetype would not be very good at that. That's perhaps the Rogue's territory.
So, the rogue can scale that wall easily, and can drop a rope down for everyone else to climb up, tying it to secure rocks and ensuring that the pitons are tight.
A wizard, even if they have spider climb, should not be as good at that as the rogue, because they have their own thing to contribute, and it's not this. Spider climb might compensate for weakness (it maybe helps you take 10 on a climb check), but it should not become a benefit (it's not better than a climb check).
Later on, when they're trying to pick out the shape-shifted doppelganger from the group of orphans, the wizard's detect magic shines, though.
Alternately, if the rogue can spot an impostor and the fighter can talk to anyone, perhaps the wizard should be able to help everyone climb that mountain.
But everyone should do one thing, and the wizard shouldn't get to do everything.
That is one thing I have noticed in these conversations is that no one seems to be talking about the wizard casting spells on the other party members to help make them more effective.
I play a wizard and I routinely cast invisibility on the party rogue to help him scout better and more safely.
Why can't the rogue go invisible without your help? That's the issue of the wizard having baked-in powers that other classes should be able to do without help.
As for knock sure as the party wizard I could memorize this spell and use it to open doors. But unlike the rogue's pick lock ability I am limited on just how many spells I get so eventually I won't be able to do it any longer. A rogue can open locks and disarm traps all day long. And that ability does not stop him from using his other abilities. He can still use his back stab in combat and evasion to avoid damage.
A wizard who has taken a lot of knock spells has to to that at cost of taking other spells that will help protect them and help them fight in combat.
Which isn't great design, either, IMO. A character needs to be able to contribute to every challenge, even if it's not their specialization. Everyone can heal a little bit (in 4e, everyone has a second wind). Everyone can strike a little bit (in 4e, everyone deals damage). Everyone can defend and control a little bit (in 4e, everyone has some method of calling attention to themselves, or afflicting statuses, or moving enemies or allies around).
A wizard shouldn't have to choose between being effective in combat and being effective in exploration. They should be at a baseline in both, even if others do better, and they are more specialized at the information part of the adventure.
Another reason spells like knock exist is to allow the party to be able to function without a rogue.
I rarely see complaints that bards and druids getting healing. Though healing is supposed to be in the purview of the clerics. But design wise they get healing so that a party can survive without a cleric.
It stops the "we have to have this class or else we can't play" and allows more flexibility to the players.
Again, this is the strength of Roles. In your example, druids and clerics both occupy the same role, so it's fine to have one without the other.
You could have a wizard occupying the same role as a rogue, but then you probably won't be in a party with both of them very often. Since the archetypal D&D party is a fighter, rogue, wizard, and cleric, I'd like to see them both contributing something different.
I keep reading that wizards step on a lot of feet but in 20 years of playing the game I have never seen this at the table. I have never seen a wizard character that takes spells to be better than another character in the group. Why would a wizard take a lot of knock spells if the party has a rogue? By doing so they limit what they can do and that is not usually fun.
What I have seen is that wizards may take a spell to help boast or back up another character that is not stepping on toes that is called being a team player and helping make the party as a whole effective.
There's enough complaints from others to suggest that this isn't a universal experience, though, and if it's causing problems, it would be smart to look at it and try to fix those problems.
Noncombat roles is one of those fixes.