The fundamental problem with modelling non mystical unarmed combat in D&D is that D&D is a game that focuses on lethal, armed battles between weapon wielding, armor wearing combatants. Unarmed combat is typically used in nonlethal battles between unarmed, unarmored persons.
This creates two problems.
If you want your unarmed fighter to fight (armed) orcs and dragons with his bare hands, you have to come up with an explanation for why he can do so competently. Realistically, a completely nonmagical nonmystical non-otherwise-augmented guy is going to have serious trouble fighting against someone else who is carrying steel.
Now, that's not a problem if what you want is for your unarmed combat to augment your otherwise armed character, and to give him a backup plan for when he's disarmed, or isn't carrying a weapon. In that case, unarmed combat works great, and has a natural role. Except it isn't a very big role in most games. Most DMs will let a player at least carry around a dagger, and once you've got that, you don't have much need for unarmed combat skills. Plus, there's little reason to add much detail to a combat style that will be rarely used.
So my personal feel is this.
Make the monk as mystical as you can. Go nuts.
Create feats available to fighters that let them fight effectively when unarmed. Something to represent some martial arts training they went through while learning all their other weapon skills.
And finally, create some hybrid feats that let a character mix armed and unarmed combat. Make it so that a person with a longsword in one hand and nothing in the other is more effective if they know a bit of martial arts. Grappling and tripping do this a bit already, but there's a lot more room for tossing in things like a snap kick to the gut of your foe during a melee.
This creates two problems.
If you want your unarmed fighter to fight (armed) orcs and dragons with his bare hands, you have to come up with an explanation for why he can do so competently. Realistically, a completely nonmagical nonmystical non-otherwise-augmented guy is going to have serious trouble fighting against someone else who is carrying steel.
Now, that's not a problem if what you want is for your unarmed combat to augment your otherwise armed character, and to give him a backup plan for when he's disarmed, or isn't carrying a weapon. In that case, unarmed combat works great, and has a natural role. Except it isn't a very big role in most games. Most DMs will let a player at least carry around a dagger, and once you've got that, you don't have much need for unarmed combat skills. Plus, there's little reason to add much detail to a combat style that will be rarely used.
So my personal feel is this.
Make the monk as mystical as you can. Go nuts.
Create feats available to fighters that let them fight effectively when unarmed. Something to represent some martial arts training they went through while learning all their other weapon skills.
And finally, create some hybrid feats that let a character mix armed and unarmed combat. Make it so that a person with a longsword in one hand and nothing in the other is more effective if they know a bit of martial arts. Grappling and tripping do this a bit already, but there's a lot more room for tossing in things like a snap kick to the gut of your foe during a melee.