Okay, so full disclosure: I just got a tooth pulled and I may not be thinking at my 100%. I encourage anyone to double check my math.
Comparing the Way of the Pistol, I actually found things to be (mostly) balanced between the Kensei and the Pistol monk. I'm going to do a side by side comparison of each of their abilities and damage potential along with any other factors I find applicable. This assumes 16 dex from 1-4, 18 dex from 4-7, and 20 dex after.
3rd level:
Kensei in melee can do 1d8+1d4+2*dex for 13 damage, or 1d8+2d4+3*dex for 18.5 damage with flurry
Pistol in melee can do 1d6+2d4+2*dex for 14.5 damage, or 1d6+2d4+3*dex for 17.5 damage with flurry.
Kensei at range can do 1d8+1d4+dex for 10 damage
Pistol at range can do 2d4+dex for 8 damage.
So out of the gate the Kensei is doing 2 more damage per shot at range, while the Pistol is doing 1.5 more damage in melee normally, or 1 damage less with a flurry in addition to sometimes having to use a ki point to reload occasionally.
The Kensei has the option to lower their damage by 2 for a 2 point AC increase, and the Pistol has the option to spread the damage between melee and ranged. All in all I'd say the two are fairly balanced between the two at this level. I could see arguments being made for either being the stronger of the two, but I think it's close.
6th level:
Kensei in melee is doing 2d8+1d6+3*dex for 24.5 damage, or 2d8+2d6+4*dex for 32 damage with flurry.
Pistol in melee is doing 4d6+3*dex for for 26 damage, or 4d6+4*dex for 30 damage with flurry.
Kensei at range is doing 2d8+2d4+2*dex for 22 damage
Pistol at range is doing 4d6+2*dex for 22 damage
Kensei gains the ability to expend a ki point to deal an extra 1d6 damage, Pistol gains the ability to spend a ki point to attempt to disarm an opponent, but only at range. Both of these are fairly situational, but I think the Pistol's is better. Kensei can alpha strike a little bit better (at a terrible cost to damage ratio) Pistol can disarm opponents, but that only works if the opponent is actually holding anything. Pistol caught up in range, and is still 1.5 more damage in melee but is now 2 damage behind with flurry damage and is more likely to have to spend ki points to reload since they're shooting twice per round now.
11th level:
The math starts to get murky here because of the Kensei's Sharpen the Blade ability occasionally granting up to +3 on attack and damage rolls and the Pistol's Zen Marksmanship ability (which I agree needs to be workshopped or scrapped)
Kensei in melee is doing 3d8+3*dex for 28.5 damage, or 4d8+4*dex for 38 damage with flurry.
If the Kensei Sharpens the blade these increase to 3d8+3*dex+6 for 34.5 damage, or 4d8+4*dex+6 for 40 damage with flurry, with a +3 to all weapon attack rolls.
Pistol in melee is doing 4d8+3*dex for 33 damage, or 4d8+4*dex for 38 damage with flurry.
Kensei at range is doing 2d8+2d4+2*dex for 24 damage
If the Kensei Sharpens the Blade this increases to 2d8+2d4+2*dex+6 for 30 damage, with a +3 to all attack rolls.
Pistol at range is doing 4d8+2*dex for 28 damage.
At this point Pistol has almost totally eclipsed the Kensei at range, and is usually better than the Kensei in melee. However if the Kensei Sharpens the Blade it still pulls ahead in damage, and has a massive boost to accuracy to boot which can't be discredited. At this point it sort of boils down to individual campaign styles and how often groups take rests to regain ki points. For shorter days or those with frequent rests, the Kensei pulls ahead. For longer days where Ki points are more scarce, Pistol is better.
17th level:
Kensei in melee is doing 3d10+3*dex for 31.5 damage, or 4d10+4*dex for 42 damage with flurry.
If the Kensei Sharpens the Blade these increase to 3d10+3*dex+6 for 37.5 damage, or 4d10+4*dex+6 for 48 damage with flurry, with +3 to all weapon attack rolls.
Pistol in melee is doing 4d10+3*dex for 37 damage, or 4d10+4*dex for 42 damage with flurry.
Kensei at range is doing 2d8+2d4+2*dex for 22 damage.
If the Kensei Sharpens the Blade this increases to 2d8+2d4+2*dex+6 for 30 damage, with a +3 to all attack rolls.
Pistol at range is doing 4d10+dex*2 for 32 damage.
Kensei at this level gets Unerring Accuracy for free rerolls, and combined with Sharpen the Blade should ensure the Kensei almost never misses. Pistol at this level gets Fatal shot, which at the cost of 6 ki points can kill a target outright, or do 10d10 damage for 55 damage. It is worth noting that this can miss and those 6 ki points are wasted.
The Pistol at this point completely trumps the Kensei in terms of ranged damage, but the Kensei can keep up with the Pistol in melee and surpasses it when sharpening the blade, and with Unerring Accuracy it has far, far better accuracy than the Kensei. I would wager that with the accuracy increases the Kensei's damage still pulls out ahead in the end in melee, and in comparable in range but I don't quite know enough about the accuracy math to say that 100% true.
My personal conclusions:
TLDR, the Pistol subclass isn't as overpowered at it might first appear, and is in fact balanced quite well against the Kensei monk. It's Zen Marksmanship ability needs to either be scrapped entirely or heavily reworked, but that's the only major flaw I could find with it personally.