The monk is a complicated case, because I really love what the Monk does right, and I'm disappointed at what the Monk doesn't do. So maybe to start off, I think we should hit what the monk does good, bad, and more mixed.
The Good
Alternative AC: This is a really solid idea that is somewhat limited by D&D's very low standard array. I think
@Mind of tempest's idea that you could cue it off other stats is a smart one (Drunken Masters, for example, might key of Charisma, while a punchier Monk might key of Strength). If you want to get wild, you might want to make the Monk's AC related to their Attack Stat as well, so you could have STR+CON Grapplemonks, STR+Dex Boxing Monks, Dex+Wis Shaolin Monks, etc. Maybe too wild, but something to think about.
Ki as a Concept: As a spendable resource, Ki is absolutely better than Battlemaster dice. It has fun stuff to do, you get more of it, etc. There are some problems with execution, but it's just a way better resource compared to their other stuff.
Martial Options: Compared to other martials, I feel like the Monks get a lot of neat stuff to work with that changes how they fight compared to other martials. A lot more focus on giving them unique combat feel rather than "I can hit stuff a little better".
The Bad
MAD Monks: Yeah, D&D's mediocre array hits monks a bit harder than others. You need Dex for everything, Wisdom for your defense, and Constitution for desperately needed hitpoints. I'd say this is more easily solved by just letting everyone get better stats, since people being generally more competent doesn't really hurt the Monk, who gains a lot more with more broadly-good stats.
So Much to Spend on, So Little to Spend: Ki is a great concept, but most things about your class are going to use it, along with your subclass. I think what makes the Shadow Monk so damn fun is that it's iconic feature (the bonus action teleport) is not a Ki-based power but simply something it can just
do. Meanwhile the Drunken Master has to spend Ki to use almost all their unique abilities since most of them key off
Flurry of Blows. Same with Elemental Monks, whose cool stuff competes with its ability to do all the other neat monk things.
Ki is also handed out stingily, with good powers costing a good amount of ki and only recovering it with a 30 minute meditation break. That can really kill the pacing of an adventure.
The Mixed
Thematics: Monks are really thematic, which is both a blessing and a curse. For starters, they get some neat ribbon abilities, but they
really focus on the Orientalist take on a martial artist, which makes it difficult to try and reskin at times. Giving the Monk some choices when it comes to their ribbon abilities would be a good way of opening up the character concepts.
The second half of this is how deeply do you want to dig down with this: Part of the appeal of 5E is to have a relatively simple system. Personally, I think more complicated martials are awesome, but as it stands it kind of runs counter to what the system is trying to deliver on. For the moment, I think I'll focus on the simple stuff, but if you want something more complex, it would be cool to look into the idea of techniques closer to spells, where you "practice" them so that you can use them each day, you can only have so many prepared, and you can learn them outside of advancement. You could even limit them by level! Not only would that give really cool class hooks (like finding new and powerful techniques from reclusive masters), but it would give a whole new level of customization. But that's like a whole work project, and I'm just not dedicated enough to really pitch that.
For the simpler ideas, I'm going with the idea that the Monk is meant to be the nimble striker of the martials, even moreso than the Rogue. So here we go.
First, two general changes.
Step of Wind now no longer costs ki. Seriously, why? Rogues can do these with Cunning Action all the time, but not the dudes who are fast enough to run on water? Given that it shares space with fancier stuff, giving it a Ki cost (when there is limited ki to begin with) just kills it as an option. Just let Monks engage and disengage as they like: they are already limited by being front-line dudes with low HP in a game where everything basically hits. Secondly,
Flurry of Blows now adds an attack to your Martial Arts bonus attack at the cost of one ki. This basically changes nothing, but allows for a class concept I want to make.
Secondly, I'm going to largely theme around a gimmick, and to that I want to make it so that each primary gimmick in the class doesn't cost ki: this way you'll still be a unique monk even when you have no ki. It also allows us to keep the ki pool low rather than having 40 ki points to do fun stuff.
Now to the subclasses. Here are my concepts:
(WARNING: None of these are terribly balanced right now, they are just sort of rough ideas off the top of my head)
Way of the Closed Fist
What is the niche?
This is meant to replace the Open Hand monk as the basic monk, and it's meant to be fast and hard. This would be a classic kung-fu fast-fists fighter or a bare knuckle boxer.
What is the gimmick?
If you hit on an attack, when you use your Martial Arts bonus action, you gain a second attack. This
includes the Martial Arts attack, so at first level you have two shots to get an extra attack if you go all out. This is also why I changed
Flurry of Blows: this class is meant to be fast attack all the way through. This might seem powerful, but remember that the monk's damage die generally trails behind the other martials anyways.
What's comes after that?
Unlike the Open hand monk, you get no healing defense nor movement control. This monk is meant to throw hands when he can, and weave out when he can't (Using
Step of the Wind, now that it doesn't have a cost). One of my ideas was that after a hit, the Closed Fist monk can spend a ki point to shift the damage die of their subsequent monk attacks by one. You could spend one for each hit, going up and up for a sort of
Fei Long Rekkakan or Mike Tyson power-flurry. I'm trying to think up an aggressive defense that isn't just "Retaliation", but if I come up with something I'll put it out there. Maybe on a hit you can spend ki to make the target's next attack against you at disadvantage? Hmm...
Way of the Shadow
What is the niche?
It's already pretty good at it, but keeping it as a tricksy ninja-fighter.
What's the gimmick?
Well, besides giving it Darkvision (Seriously, it's handed out like candy, might as well give it for this one), as a bonus action, instead of extra attacks you can gain a Sneak Attack bonus like a Rogue, with all the necessary restrictions (be at advantage or have them be next to an ally). This fits the archetype better, moving away from a lot of attacks to a few
solid ones, and gives it an interesting possible tradeoff depending on how its implemented: You could have them do it before they make attacks (which creates a tradeoff) or afterwards. I suppose it depends on the Sneak Attack scaling, which I would probably put behind the Rogue but still be effective. This would also sync well with their powers, which allow them gain advantage in a number of ways.
What would come after?
As a defensive power, allow them to use their teleport as a reaction after getting hit: you take the hit, but you can Naruto-Log trick out of there (if the conditions are correct for the teleport). You could maybe make this have a ki cost. But honestly the Shadow Monk is already fairly good. I might make "See through your own darkness" Darkvision be something they can get as a Ki point or two option, so that every Shadow Monk doesn't just go into Warlock to get that (incredibly useful) invocation.
Way of the Drunken Fist
What is the niche?
This is the defensive monk, based on redirection and avoiding damage.
What is the gimmick?
They can Dodge as a bonus action. Their previous power is basically made obsolete by taking the
Mobile feat, which... look, I know feats aren't necessarily the base game, but they certainly feel like it within the D&D community. This is powerful, but at the same time you can limit the monk by making their powers more defensive in nature, which is risky because if you
do get hit, you don't have the hitpoints to recover.
The second half of the gimmick is that if the opponent has Disadvantage and both its attack dice would normally miss, the Drunken Master can use their reaction for an opportunity attack. Not a free reaction attack, but with Dodge (or clever situational stuff) you can potentially get some use out of it each turn. This makes for a less aggressive, more reactive fighting style.
What would come after?
Redirect for a ki point is already good. I'd say a later level one would be if an attack misses, you can spend a ki point to recover your Reaction: you can't use it on the attack that happened, but misses essentially reset your ability to cause damage.
Way of the Elements
What is the niche?
Controller monk. Move people around, slow them down, modify the battlefield.
What's the gimmick?
As a bonus action, you can shift your basic attacks to be elementally-based. This change would stay until you changed it with another bonus action or you released it as a free action.
- Fire: Deal fire damage, maybe half your proficiency in extra damage? One point of extra damage for every hit scored after the first? Dunno, fire is pretty useful until it isn't and having ongoing burning damage would be a hassle.
- Water: Cold damage, and target has -10 feet of movement (does not stack).
- Wind: Bludgeoning damage and the target must make a Strength save or be knocked back 15 feet.
- Earth: Bludgeoning damage and the target must make a Dexerity save or be knocked prone.
I might consider putting lightning in there, but you get the idea. It takes the Open Hand movement abilities along with some elemental damage.
What would come after?
Elemental cantrips at different levels, particularly the ones that are less focused on combat to give them some of the basic shaping stuff. Give them some "focus attacks", basically elemental attack that focus less on damage and more on changing the battlefield: For example, an ice attack that hits one person but then spreads ice in a small area to create the possibility of people tripping. Keep elemental disciplines, but lower their outrageous ki costs and try to create more unique powers and fewer reskinned spells.
This subclass would probably require the most work, given the options you can give it.
Way of the Grappler
What's the niche?
Well, obviously grappling! Controlling individuals, compared to the broader abilities of the elemental monk, as well as higher damage output and a bit of tankiness.
What's the gimmick?
Okay, so this is off the top of my head, so it's probably not going to be balanced at all. But my immediate thought would be a "Slam" action. It would work as follows:
- An opposed Athletics Check. I'm kind of torn on this, but it feels like it fits the bill closest. It's also the riskiest, since opposed checks are generally more prone to failure.
- On a success, you shift your martial arts die up by one and cause two dice plus your Strength modifier worth of damage. This is part of the big tradeoff.
- Both you and the target are left Prone. You may also choose to move them to any square adjacent to you.
- Slams also end the grapple.
This creates some level of risk for the reward. A second power is requires to stop you from getting hammered after making a Slam, which I'm going to call "Kip-up":
- As an attack, you can instead stand up immediately.
- As a reaction to an attack, you can stand up immediately. (I'm torn on whether this should be before or after the attack. I prefer before, given the Monk's squishiness.)
This means that this monk won't stay on the ground long, but it's also got a complicated set of actions to work with: the monk can get up to avoid damage, or they can stay on the ground and potentially grapple again for a ground pound situation. Again, this is all off the cuff, but I think it kind of works in concept.
What would come later?
Obviously a way to give them Expertise: Athletics, otherwise you're going to get a lot of immediate dips into Rogue. But this class has a bunch of fun ways to modify it:
No Sell could allow you to spend ki to get resistance against a single opponent's attacks for their turn, you could spend ki to get more damage dice on your attack (SPINNING PILEDRIVAH~!), you could also spend a ki point to restand you and not break the grapple (Rolling Germans), toss your target further (Overhead belly-to-belly), or even transition into a restrained hold (Flying Cross Armbreaker). Whole bunch of cool stuff here that would obviously need to be balanced, but could totally work. And you could maybe have the option to have different move key off Acrobatics for more of a high-flyer/lucha flavor.
So yeah, there you go.