Disadvantages of this build:
* Hex is all attacks where Hunter's mark is only weapon attacks (will not work with your bonus attack and flurry).
* With this build I prefer the Horde Breaker/Mobile combo for more mobility.
Colossus Slayer also only kicks in once per turn. So Hex > Hunter's Mark+Colossus Slayer, especially when you pick up Extra Attack at Monk 5. The difference is +3d6(+1d6 with Extra Attack) Hex vs +1d6+1d8(+1d6 with Extra Attack) HM+CS. Hex has the minor benefit of giving disadvantage on certain ability checks/saves too, and can also squeeze another 1d6 of damage if you can attack as a reaction, such as with OAs. So Warlock > Ranger.
I like the idea of Monk/Warlock, but when you add in Rogue for the Sneak Attack dice, you lose a lot of Ki, so you're not even gonna be able to Furry of Blows every round. I don't know about you, but I usually get in more than 6 rounds of attacks between Short Rests, so stopping at Monk 6 for Shadow Step isn't optimal.
But I don't even think Ki limitation is the only problem here. The real problem is that Shadow Step, Hex, Flurry of Blows, and Cunning Action (to Hide) all require bonus actions, meaning you can't do all 4 on the same round (such as a surprise round for Assassinate synergy). You can Hex+Assassinate on the first round, and then Flurry of Blows with Hex already up to keep the damage up after the first surprise round nova, but that's it. And the first round isn't really even a nova because you lose out on Flurry of Blows. And you can never Shadow Step or Cunning Action Hide anymore if you're using Flurry of Blows. This build is a mess.
Dropping Rogue would be best, because this build is trying too hard to be a nova burst build when Monk/Warlock is clearly meant to be a DoT build. Hex+Flurry of Blows is wonderful together. Just stop there though. Or take other DoT boosts, like 3 levels of Ranger to throw in Colossus Slayer on top of that. But whatever you do, stop trying to build around nova. Take Warlock to 3 for a Sprite familiar that can fly invisibly next to enemies to proc Sneak Attack 100% of the time for every round too, because you're not going to be giving up Flurry of Blows for Cunning Action Hide to reset each round. Way of the Shadow is pointless because you're never, never, never going to Shadow Step when you have so many better options for your bonus action. Go Way of the Open Hand and throw some stuns in there or Sun Soul+Undying Light for some ranged synergy. Way of the Open Hand lets you prevent OAs too, so if you decide to run off after attacking, you could. Or better yet, take the Mobile feat, knock them prone AND then run off without provoking an OA. That way you don't risk losing concentration on Hex. You could always just stand there too. You have high Unarmored Defense AC, Uncanny Dodge for half damage or Deflect Missiles. You're a Monk, Harry. You've got some tricks up your sleeve to boost your damage, sure, but you're not a burst class. Stop trying to be a hit and run Rogue.
Monk 11 is ideal. Not for the Monastic Tradition feature, although that's nice too. 11 gives you d8s, which you'll be using 2 of through Flurry of Blows almost every single round, regardless of Sneak Attacks or whatever. Monk 6 or 8 might work too, but I'm concerned about having enough Ki for Flurry of Blows. I'll do the math below.
Even on a Surprise round, I would put Hex up instead of Flurry of Blows. Why? Because Hex damage is doubled too from the auto-crit. Let's do the math. Attack gets 1d6*2 + Extra Attack gets 1d6*2 + next round you can Flurry of Blows instead of casting Hex means 1d6*4 from 4 Hexed attacks = 8d6 = ~28 extra damage by the end of the Surprise round and first full round. Assuming Monk 11, if you Flurry of Blows instead of Hexing on the Surprise round, then 2d8*2 from the bonus action unarmed attacks + 2d6 from 2 Hexed attacks but no Flurry of Blows on the first real round = 4d8+2d6 = ~25 extra damage by the end of the Surprise round and first full round.
Hex beats Flurry of Blows, even with Assassinate on a Surprise round... More proof that you should be building for DoT, not burst.
Let's look at the possible builds. The minimum we need is Monk 5 for Extra Attack and some Ki to work with, Rogue 1 for Sneak Attack or 3 if we want Assassinate for burst..., Warlock 1 for Hex though 2 will give us Invocations, and possibly Ranger 3 if you want the Dueling Fighting Style and Colossus Slayer. Have to use a Finesse Monk weapon to trigger Sneak Attack, so Shortsword. Shortsword works with Ranger's Dueling style too:
#1 Monk 11/Rogue 6/Warlock 3 gets 3 ASIs, 3d6 SA dice, 4 skills with Expertise, Uncanny Dodge.
DPSR@20 (Hex as BA) = 2d6+5+2d6+6d6 + 2d6+5+2d6 = 59
DPR@20 (FoB as BA) = 1d6+5+1d6+3d6 + 1d6+5+1d6 + 1d8+5+1d6 + 1d8+5+1d6 = 60.5
#2 Monk 11/Rogue 5/Warlock 4 same as above, but drop 2 skills with Expertise for an extra ASI, an extra Spell Known, and a Cantrip.
DPSR = 59
DPR = 60.5
#3 Monk 11/Rogue 4/Warlock 5 gets 4 ASIs, 2d6 SA dice, 4 skills with Expertise, no Uncanny Dodge, 3rd level Warlock spells, and another Invocation (can take One With Shadows to basically be a Way of the Shadow Monk while still going Way of the Open Hand). 1 less Sneak Attack die lowers our damage slightly.
DPSR = 52
DPR = 57
#4 Monk 12/Rogue 4/Warlock 4 Cross between #2 and #3 to end up with 5 ASIs! I don't think you need that many, but more feats can't hurt. You're MAD as it is, so a higher CON is a good thing.
DPSR = 52
DPR = 57
Or if we add Ranger in there (getting MAD, still assuming 20 DEX though, good luck with that):
#5 Monk 11/Rogue 3/Warlock 3/Ranger 3 gets only 2 ASIs...but that might be all you need to cap DEX. By RAW and the multiclass spell slot table, you only get 2 level 1 Ranger slots because you're considered multiclassed. The bonus here is Colossus Slayer and the Dueling Fighting style. Still using a Shortsword.
DPSR = 2d6+5+2+2d6+4d6 + 2d6+5+2+2d6+2d8 = 65
DPR = 1d6+5+2+1d6+2d6+1d8 + 1d6+5+2+1d6 + 1d8+5+1d6 + 1d8+5+1d6 = 65.5
Or if you insist that 6 Ki points (and therefore 6 Flurry of Blows) per short rest is enough:
#6 Monk 6/Rogue 7/Warlock 3/Ranger 4 has 3 ASIs, but puts out the highest numbers (as long as your Ki lasts). 3 ASIs is the minimum IMO, because you probably need 2 for DEX boosts and a 3rd for War Caster (important for maintaining concentration on Hex). Took Ranger to 4 instead of Rogue to 8, because it gives you an extra spell slot (which oddly enough, Warlock 4 won't give you).
DPSR: 2d6+5+2+2d6+8d6 + 2d6+5+2+2d6+2d8 = 79
DPR: 1d6+5+2+1d6+4d6+1d8 + 1d6+5+2+1d6 + 1d6+5+1d6 + 1d6+5+1d6 = 70.5
#7 Monk 6/Rogue 6/Warlock 4/Ranger 4 for the same as above, but drop 1 Sneak Attack die for an extra ASI. A fair trade. CON or Resilient CON would be nice.
DPSR: 2d6+5+2+2d6+6d6 + 2d6+5+2+2d6+2d8 = 72
DPR: 1d6+5+2+1d6+3d6+1d8 + 1d6+5+2+1d6 + 1d6+5+1d6 + 1d6+5+1d6 = 67
#8 Monk 8/Rogue 6/Warlock 3/Ranger 3 is the same as above, but you drop 1 ASI and a Ranger spell slot for 2 more Ki. As stupid as this sounds, this might be the best compromise. You still have 3 ASIs to cap DEX and grab a feat like War Caster for maintaining Hex.
DPSR: 2d6+5+2+2d6+6d6 + 2d6+5+2+2d6+2d8 = 72
DPR: 1d6+5+2+1d6+3d6+1d8 + 1d6+5+2+1d6 + 1d6+5+1d6 + 1d6+5+1d6 = 67
tl;dr Build DoT, not burst. Throwing Ranger 3 in boosts damage a bit. 6 Monk > 11 Monk, but you have to Short Rest as much as possible. 8 Monk is a possible compromise. Builds #6, 7, and 8 are best, depending on how many Short Rests your group takes as well as how many ASIs you want/need.
