Ok, Im going to run a few scenarios methink this weekend and see what happens, but here is my current thoughts.
1) many have emphasized ways attackers can bypass the walls using magic. Most of these can be safeguarded against magically, but would be VERY expensive. However, castles in and of themselves are expensive projects, adding in the cost of some magical protections may not seem so bad by comparison. Additionally, hiring a mage good enough to give you an army of invisible flying attackers will be JUST as prohibitively expensive, so I feel the cost question is moot.
2) Assuming parity of firepower, both mundane and magical, what is to prevent the defenders from staying hidden behind their walls and waiting for the attackers to waste spells bypassing them. Yes, a mage may be able to disintigrate a section of wall and send in a bunch of invisible troops that have been enhanced. However, he is now down key spells, while the defending mage is still at full power and can deal with the troops as they enter the breach. My point is that the walls force a depletion of magical ability from the attackers, who will not have any real advantage just because they have breached the walls (they will have mearly NEGATED the advantage the walls gave the defenders). Even disregarding the fact that even once the walls are overrun, the defenders can use ambush tactics and known terrain to give themselves an advantage, the disparity between the available magical firepower between the two forces now gives the defenders an advantage (whether or not it is as large as the original wall advantage could be argues, but the defender still lokos to be in the better position).
Assuming parity of force, I can not see any way that anyone can give a definative edge to the attackers. To do so, one would have to claim that having a fortified position is somehow a liability (or that attacking such a position is an advantage).
edit:
As for "ninja" style raids, castles are not set up for this type of conflict. But I somehow don't see groups of high level adventures (which would have to be the case to pull this off) generally being available to do this type of work, at least once again not at anything resembling a "reasonable" fee. If your thought is "can a castle stand up to a group of high-epic level PCs" then I would answer most likely not. But then again, the same could be said for most anything.