Well how how about protecting the wizard niche? A rogue at high levels who has poured ranks into use magic device can very effectively compete with a wizard in the spell category. Not only can he use arcane spells but he can also use divine spells.
The rogue cannot come even close to the wizard. He can sub in a pinch, at great expense (non crafted wants and scrolls are expensive!) and only at the whim of the DM (who must actively give access to the proper magical items).
Meanwhile the mage can step on the rogue's niche as an afterthought (knock, invisibility, silence, scrying etc.) and (with crafted scrolls) still have plenty of room to do his own thing.
The playing field is not remotely equal.
The rogue also with his back stab can compete with the fighter when it comes to doing damage in combat.
Again only at the whim of the DM. Backsab is extremely situational and often does not apply (undead, constructs, slimes etc.).
The wizard, on the other hand, by mid-high level can summon monsters that give a fighter a run for his money and again still have plenty of oomph left over for other things.
Then there is the cleric with his buff spells and access to heavy armor he can more than compete with the fighter to be the frontline tank.
I've noticed this only applies if there are 2 (or more) clerics in the party. Otherwise the cleric is so busy healing (and swapping spells for healing) that "kicking butt" is tertiary. On paper the cleric seems more powerful, but in play this always seems balanced by his almost indispensable medic role. But even here caster vs. non-caster, that is the real distinction.
It is almost impossible to totally protect niches there will always be some overlap.
The big problem arises when the niche encroachment is extraordinarily one sided. The mage can fill in for the rogue with some inconvenience; the rogue cannot fill in for the mage, not really.
As for the spells you mentioned like spider climb it is should be in the wizard repertoire. Wizards are not as strong as fighters and they don't have climb as a skill so they need to be able to climb that cliff face. They can also use it to help other classes.
This is why I love 4e rituals, they solve this problem. Sure the wizard can open the lock just like the rogue. BUT the rogue can do it in seconds while the mage takes 10 minutes, it's pricey, loud, and it fatigues the mage (costs a healing surge).
As for knock sure as the party wizard I could memorize this spell and use it to open doors. But unlike the rogue's pick lock ability I am limited on just how many spells I get so eventually I won't be able to do it any longer. A rogue can open locks and disarm traps all day long. And that ability does not stop him from using his other abilities. He can still use his back stab in combat and evasion to avoid damage.
A wizard who has taken a lot of knock spells has to to that at cost of taking other spells that will help protect them and help them fight in combat.
A scroll of knock is not that expensive, there are rarely that many must open locked doors where a scroll or 2 will not sufice. And if there are - use a wand; I don't believe I've seen 50 must open doors in 20 levels of adventuring!
Another reason spells like knock exist is to allow the party to be able to function without a rogue.
I have seen several parties function without a rogue. I have yet to see even 1 party function without a wizard (though I'm told they exist).
I keep reading that wizards step on a lot of feet but in 20 years of playing the game I have never seen this at the table. I have never seen a wizard character that takes spells to be better than another character in the group. Why would a wizard take a lot of knock spells if the party has a rogue? By doing so they limit what they can do and that is not usually fun.
What I have seen is that wizards may take a spell to help boast or back up another character that is not stepping on toes that is called being a team player and helping make the party as a whole effective.
I don't remember near the niche encroachment in 1e/2 but have seen it since 3e and I firmly believe (as I've stated elsewhere) that extremely easy access to scrolls (it's a free feat) and wands (also essentially a free feat) is the big culprit.