Dungeons & Dragons 3rd Edition had a great concept: core base classes upon which all characters could be built. If you want to play a certain kind of fighter then take different feats than that other type of fighter. If you want to play a Fighter/Magic-User then just multiclass 50% in each and no XP penalties apply.
Two problems arose:
1. Multiclassing under 3rd edition just didn't work for most class combinations where one or both classes didn't rely on a weapon and Base Attack Bonus. Fighter/Rogue worked great. Wizard/Sorcerer was a terrible choice.
2.
D&D is a class system. The build-it-yourself character concept requires a lot of flexibility that is hard to achieve with only a few classes. Also, feats were too few and far between to provide a satisfying level of distinction between class builds.
Prestige classes were the perfect solution. The most useless prestige classes weren't available until you were 11th or even 15th level. By then the game was practically over for 90% of games/players. Prestige classes allowed game designers to take a step back and fix the two issues above with something that players seemed to actively enjoy (at least based on sales of books that contained prestige classes).
Why did prestige classes resonate so well with players?
I argue that people don't play
Dungeons & Dragons because it is the best game, nor the most realistic game, nor the coolest game.
D&D continues to win the RPG war because it is simple. Character class progression is the largest part of
D&D's simplicity. Once you choose your character's archetype your at least 75% done and you don't have to fiddle around or master the rules to get a character.
Prestige classes are popular because they're conceptually simple. Many prestige classes are very targetted character builds or even narrow builds within a base class. It is easy for player to grasp a build when it comes pre-packaged for them in a tidy prestige class.
Furthermore, designers felt they had the power to give out more powerful yet specialized abilities to prestige classes that weren't available through multiclassing or feat chains. Therefore prestige classes became a very useful and successful way for players to reach their character concept builds and HAVE FUN!