If you are just a paladin yourself, its hard. If you have other party members, like a wizard, its easy.
Build a 'dungeon' around said portal. The traps should simply be millions of cheap, permanent magical traps. (Explosive runes, Sepia Sigils, Symbols of Insanity/fear/weakness/death, Glyphs of Warding, etc) Build this 'dungeon' as a one-room thing, fill it with the nastiest magic traps you can. (Permanent Invisible Prismatic Spheres around the Portal work wonders) And further complicate things with mechanical traps like poison gas, maybe a Golem of a nasty nature.
Now, after all that is done, build a huge thick stone wall/dome/barrier that completely encases it. Now encase that in lead. Now put more stone over it. Now get ye olde wizard to move earth/stoneshape/etc to hide the dungeon inside a mountain. I didn't say anything about an entrance, so no pesky adventurer is going to poke their noses in, Lead blocks divinations, so no one will know it's there should they look. And if they do know where it is, if the try and teleport in, even if they don't mis-target, the traps should catch them unaware.