By the looks of it, I'm pretty flexible with what the players can be and do, so long as if they add to everyone's fun, and so long as if I can work with it.
But even then, I have a few archetypes I won't allow.
1.
The Munchkin
The term Munchkin comes from a term used by older gamers to talk about the new gamers who "play RPGs to win". The term Munchkin means that the new players are "Short and silly".
But to go deeper into detail, those players invest absolutely no dimensions to their characters, other than min-max their stats for combat. Then when they kill everything in a dungeon, they are the first to say before anyone else, "I take ALL the loot!

"
2.
The Pointless One
You know those players who in one way or another just don't end up doing anything? Yeah, its those people who either sit back as part of the audience (Which could be a problem with coming out of the shell, which I can forgive once I understand that), or make a character that literally is useless, and is played as such.
Like when the PC runs, the players will say that they trip and fall over. Or when they swing their sword, they'll miss and fling their sword into a wall or something. Its to the point where its just borderline griefing, which those concepts might possibly work for NPCs, its just plain annoying to handle when a PC deliberately does that.
This takes me to number 3.
3.
The Griefer
Ok, so this person's play style in the party is to cheese off the other players, and make it less fun for them. I ask them the question, "Why do you do this?" Then they'll give an answer thats usually something like, "Its just a game, don't take it so seriously", or "I don't like the party", or something stupid like that.
To respond to the first answer, yes, it is just a game. And a game is an event purely to assist with the enjoyment of others. The problem is, no one is enjoying themselves when they go and do whatever stuff it is that keeps players from enjoying themselves. I ask the griefer, "What is the point of what you're doing? Its obvious that you don't want to work with others, and its obvious that you don't want to play this game. So do both you, the group, and me a favor, and get off my table, and find something else to do".
4.
The Mary Sue
The difference between a Mary Sue and a Munchkin is that a Mary Sue is a character that looks and acts like a "Speshul Snowflake with Speshul Powers", whereas a Munchkin seeks to win at RPGs.
The Mary Sue is a character that, despite all the labor poured into him/her, still has absolutely no depth to them, relying only on whatever they wrote down for their appearance (No doubt having purple hair as a human, or having some kind of an exotic pet like an ooze that changes colors depending on the mood or something), and then they walk around like the story rotates around them and only them, and all others, including the other PCs are just their butlers or something. Outside the obvious problems with that point of view, its also bad that such a story has to revolve around such a boring character.
But really, I consider myself a flexible DM, and will allow stuff outside the PHB so long as they consult me on it. Heck, I even allowed a player to play a flesh golem after we talked about how this flesh golem was not going to be immune to spells, and will be a living construct, and having a mind.
That flesh golem is now adding to the experience for all the players and me as well.
That aside though, when I see the above 4 examples, I try to stop the problems before they get approved. The Mary Sues will be given more depth, the Griefers will be kicked out, the Pointless Ones will be encouraged to assert themselves more often, and the Munchkins... oh boy do I have a safety net for them.
When I sense Munchkins, I try to talk to them about it, and to use the backstories I made them write as a guide to help them to make their characters more interesting to everybody.
Then when they try to loot everything before everyone else, I ignore any cries of, "I loot everything!" I instead manually split the loot, the gear, the money, everything between each of the players. Then if they try to loot everything while the players are either everywhere else, or still fighting the dragon, I tell them, "You cannot loot until the dragon has been defeated". I know it makes no sense, but nor does magic, or Elves, or Dragons for that matter...