For me, the thing that would make a Prestige Class worthwhile is if it were truly Prestigious. Like, the Archmage example is good, because you can't just be an Archmage at 1st level. Similarly, things like High Priest, Guildmaster, Grand Champion, and cough Warlord seem like they should be reserved for higher levels.
However, for all of these, the question is: Do you need the full mechanical weight of a class or do existing mechanics cover it? Like, what's the difference between an Archmage and a high-level wizard? Or between a Warlord and a high-level fighter? Could you become a Guildmaster just by, you know, being the master of a guild, in-game? etc.
They certainly would need to be
prestige classes. There's two word in prestige class, and both are important
A prestige class is meant to be a
class because there is a
progression inside it. This is where it is fundamentally different than a feat, although there used to be feat chains that also were a good idea (and just as well as PrCl, they got a bad implementation).
But then it is meant to be
prestige exactly because it's something that should come up only later in the game, but even more importantly because it should
not be available to everyone. The 3e prestige classes could generally be taken at 5th level or later, and even that is a bit too low to call "prestige", but we have to understand that this was probably chosen because (a) players were not always supposed to plan ahead and gather requirements to qualify as soon as possible, and (b) most games don't reach 10th level and making prestige classes too high in level means few players will ever get them.
As I said, personally I am pretty confident that people hate 3e PrCls because both the designers and the players largely abused them, not because the concept or mechanic is bad.
The players abused prestige classes by stacking lots of them, cherrypicking a few levels from each because some prestige classes were front-loaded, and regularly disregarded the narrative nature of them: this is pretty much because of the widespread competitive culture of 3e games and the "character builds" sub-hobby, but hopefully both those are not nearly as strong in 5e as they were in 3e. Despite sometimes having costly requirements, 3e PrCl did not feel prestigious because most DMs were making them available to everyone, and players demanded that if they had spent money on a book then they must be given the right to use whatever was in it.
Designers also abused the idea because gamers were asking for more and more PrCls at each new book, so obviously it became a milk cow... which meant that designers were constantly trying to come up with new, extremely narrow character ideas, and make a PrCl out of each... but then, because class design is hard work, they often ended up with 10 levels which really contained 2-3 new abilities at most (and often re-hashed mechanics, slightly modified). There were even examples of truly bad PrCls that had literally zero unique features, and were only "+1 spellcasting level" and a few bonus feats. Finally, they even exploited the PrCls mechanics for "fixing" what was regarded as a bug in the core rule i.e. multiclassing spellcasters yielding insufficient spells abilities.
All this crap doesn't have to happen in 5e.
I would argue that neither "Archmage" or "Guildmaster" are prestigious and specific enough to be prestige classes however. They are too generic to be forced into a single class.
Instead, it might be better to consider the prestige class approach to represent a character concept that has the following properties:
- it represents very restricted knowledge: it's either only available to members of a closed group, or to a "chosen" of a certain kind
- it's absolutely unique in terms of abilities acquired (I would avoid ANY non-unique features in a prestige class progression)
- those abilities should be capable of standing on their own, not "boosts" base class abilities, so that the prestige class is potentially a good thing for every character, but then the prestige class works just like multiclassing and therefore a PC can always choose to level up in the original class
- it's a fairly long path, no less than 5 levels worth of abilities but the longer the better, so there is no way to fit this into feats or feat chains (I don't think it's a good idea to have feat chains longer than 3 feats, given the 5e feats acquisition rate)
- it's a path that can be taken potentially at any time during a PC's life (not at a specific time like subclasses or backgrounds), because you should not be able to plan it*
- better to have narrative requirements only for the previous reasons: to be accepted into the closed group you need to prove yourself through actions, not to produce a CV of feats and skills...
*of course the
player and DM can plan it, but the PC should not be able to think of it like planned education, because in order to be really
prestige it has be something by invitation-only, and possibly even secret, so a PC wouldn't even know that the option exists until invited or chosen
For example, the original
Shadowdancer prestige class could be a good candidate, because the character concept is "all about shadows", but doesn't necessarily gravitate towards fighting, exploring or spellcasting. It can include abilities of stealth, magical movement or teleportation through shadowy areas, creating illusions with shadows, summoning shadow companions for help or even combat aid... all of which can be useful to everyone, in combat or other pillars. And all of which can have a progression, so it makes sense to spread them into levels.
I think a good test for a character concept could be:
if the whole party would gain this prestige class, would it be good for everyone? (note that this doesn't mean everyone should stop levelling up in their base class).