You create a new class when the reduction in effort on the player's part to create this class variant, combined with the popularity, and thus frequency, of the concept, outweighs the cost of development, balancing, and space within the physical book itself.
You have paladins because constructing a fighter to match what a paladin is, is too much work for too many people, and enough people want paladins that you know a ton of people are going to gripe about it.
You don't get mystic (yet) because the cost of constructing an entire new system for psionics, which isn't shared with any other classes, and adds a ton of weight to the book size, outweighs the relatively small population of players that want that play mechanic.
~
You start with Adventurer. Adventurer needs skills and magic and a combat system. Theoretically, the Adventurer class can be anything. Practically, it's not a functional starting point for a regular player, though.
So you break it apart into those who specialize in physical combat, and those who specialize in magical combat. Everyone uses skills, so you won't worry about that. And for simplicity, we'll call them Fighter and Mage.
This generally works, but is a bit bland, and you've barely scratched the surface of the different ideas for character types that players have had. You've got plenty of space left in the Player's Handbook, so you decide to expand a bit more.
Out of Fighter you carve the big beefy barbarian, the holy defender, and the sneaky thief type. Barbarian, Paladin, Rogue. Lots of people like these themes.
Out of Mage you carve out the holy priest, the studious researcher, and the guy who made a pact with the devil. Cleric, Wizard, Warlock. Lots of people like these themes.
You still have room, so you look a bit more. People like woodsy/pagan/back-to-nature stuff. You can put together a Ranger on the Fighter side, and a Druid on the Mage side.
Then you have those who were on a bit of a superhero bent, and they want a caster that just has natural magic. Sure. Sorcerer. But if you put that in, the people watching kung fu movies will want to bring in their Monk. So you say, sure, you can make that work.
Then someone comes in and says, "You know, we really need a Bard. There's way too many people that want a bard."
At this point you've kind of reached the limit of the number of pages you were allowed. And you go back and forth on what's really necessary to keep. You finally get convinced to OK the Bard, but you have to drop one of the other classes, and decide that the generic Mage really contributes the least, while you still need your Fighter for random NPC town guards. Plus, abstractly, you've kinda covered the gamut of what you need on the magic-using side with the existing classes (magic via research, innate ability, pact, and whatever the treehuggers do). Heck, Bard doesn't really add anything mechanically unique on the magic side.
~
So now you've added as many classes as you can fit in the book. 12 classes is already pushing things in complexity, too, number-wise. Any new class is going to have to prove its worth in page cost, or be popular enough to push one of the other classes out.
And then players come back in and say, "You know, you don't really need all these classes. It should be possible to derive them from a few simpler classes." And you say, "You know what, you're right. We don't need them. But they're really useful to a whole lot of players, especially the new ones, and we're aiming for ease and convenience for as many people as possible."
Then another group of players asks, "Where's Mystic? Where's Warlord? Where's Witch? Where's Shaman? Where's Artificer?" And you say, "Sorry, we don't have room for them. There's an actual, physical limit to how much we can put in the basic system. We love the ideas, but we'll have to wait til we can put in a dedicated supplement for that type of material. Til then, there's lots of ways to build what you want; the system is supposed to be flexible. Just ask your DM."
~
And thus, the (fictional) story of classes in D&D.
