Well, if you're guarding the castle as a wizard, you presumably guard the castle in the same way as any other guard: By patrolling the walls looking for intruders. Watching from a high tower works also. Being a wizard, you should focus on detection spells to enhance your ability to detect intruders attempting to approach the castle. True seeing is a good choice of spell.
If you mean DEFEND the castle as a wizard, you take up the role of counter-battery artillery, and destroy any enemy siege equipment being erected to fire at the castle. You may also be called upon to perform the above role of spotting invisible intruders. Dispatching magically transporting intruders, however, is a job for internal security people, as wizards don't do well in close-quarters combat like the sort you'd run into inside of a castle, unless you're willing to inflict a large amount of collateral damage to the castle itself, which I doubt.
If you're trying to BUILD a castle, and your expectations are to be attacked by small adventuring groups consisting of high and mid-level characters, rather than armies of mooks with a siege train, forget about castles, and go with a classic simple building. Since such people generally have means at their disposal to easily demolish any structure you can conceive of by means of magic, you're better off not tempting them to do so. A simple hut will suffice. With no normal people around for a thousand miles, siege is somewhat unlikely, and a high-level group will likely demolish anything you can make, messily.