Yeah, this is right. But still, scrying someone generally lets you get a "viewed once" level of familiarity for the purposes of teleport. Scrying speficially states that you can see the subject's "immediate surroundings," within about a 10-foot radius. Even greater teleport requires that you at least have a description of the place.
This means that once your character can start tossing around scrying and teleport spells (somewhere around 9th level), the formula for a villain-motivated adventure generally breaks down to something resembling the following: 1) Scry the villain to figure out where he is; 2) teleport in and start wailing on him. If the scrying fails there's generally no reason that you can't just sit around and wait until tomorrow to try again.
Even giving a villain access to detect scrying usually doesn't help too much. If he detects someone looking in on him, he's likely to return the favor and come to the PC's instead of the other way around (he'll probably at least send some minions). The specifics vary from character to character of course, but the general trend seems to stick. You can only have so many fortresses with permanent Mordenkainen's private sanctum effects up, after all.