Prestige Classes for me are highly (player) difficult Sub-Classes that require multiclassing to middle or high class levels to even qualify for.
Being so they are even more highly defined, and therefore limited in scope, than a standard Sub-class, much less a Core class.
There are no design examples that find that definition from 3.x that I know of. Bards from AD&D do, so here's an explanation based on their design:
Bards are members of a rare organization of highly trained fighter/thieves that have been recruited and trained by another specialty organization, the Druids of the world. These are campaign setting defining classes, so having Bards means Druids must also be part of the world or they need to be redefined.
Bards are a unique Cleric (Druid) sub-class that focuses on the natural divine ability of music, song, and dance. They are trained as Druidic spellcasters, but performed within the aforementioned narrow range of casting style (something any common Druid could choose too, but Druid characters have a broader range of options within which to grow).
What do you gain for all of this added pre-definition, if other classes have the option of growing into it too?
For starters, it takes broader classes longer to advance to a point where they are (almost) as capable in a field as those who simply began and stayed within it. Any Fighter can take oaths, gain followers, trade for holy powers, and eventually become a Paladin. In fact, that's how the Paladin class likely came about in the first place. But anyone who is trained by a Paladin from the start as a Fighter sub-class begins play with those options.
The Bard, by contrast, requires player proficiency in both the Fighter and Thief core classes and the Bard class must therefore cover in its scope some degree of each of those class's breadth of play. Beyond that they advance within a smaller scope of the Druid sub-class as well. No core class or sub-class alone could ever be defined by its player into a prestige class due to its unusual breadth of requirements. It's a very niche field and as such it has a lot of peculiarities to it that wed it to the setting. Meeting a Bard, even a low level bard, becomes a rare and dangerous occurrence.
I should also mention, level 1 for any Prestige Classes begins with zero XP in the class, but due to their proficiency requirements and starting difficulty they are a higher level challenge as foes to those in the base game than Class Level would normally denote. That is beyond the standard multiclassed levels already affecting the challenge.
To make a new prestige class I would suggest going along similar lines as the Bard. An Assassin 5 / Illusionist 5 might be the class advancement prerequisites for a secret sect of death worshippers [Dreamslayer?] who wish to release the living from "the dream of reality" or some such craziness. Trained by a evil Illusionists, this prestige class might continue to gain within a smaller scope of Illusionist spells (some specific to it) and still persist in its skulking and disguise training and its focus on killing. Perhaps they eventually are trained by Dream Demons or some such to invade dreams and instill waking terrors via phantasms.
In the end though, all prestige classes and sub-classes are pretty much optional. Something specific to each campaign setting and is best offered incorporated in such.
