Yes there are a lot of assumptions, but when we are talking about applying advantage on some attacks, but only when you hit, and not on others you are going to have to have a lot of assumptions.
Ok a couple things. ... you make some good points, and I did assume the guy with pistols was not in melee. But it is not as easy as you think. If you hit with my pistol on my 2nd attack (most likely with advantage) then my first attack of the next round has advantage because of Vex. So it is not at disadvantage, it is straight up and if I hit with that one the second is not at disadvantage.
Also I can transition to shortswords and get advantage on those attacks and still do more base damage than a vex-nick combo.
Right, and while I understand you cannot actually mathematically account for every single variable, you need to explicitly state these kind of assumptions. Because making the claim "Dual-Wield pistols with vex always do the most damage" is very different than "Dual-Wield Pistols with vex do more damage if you assume always being within 30 ft, but never being in melee and if you do end up in melee you can switch to a different weapon to keep the advantage"
It does get very complex, you don't really have DPR with Vex because the advantage goes into the next round and because it triggers on a hit each attack is not independant of other attacks. Your average DPR actually goes up every round. Having advantage on the attack I am making currently (because of Vex on the last attack) makes it more likely I will have Vex on the next attack I make because of Advantage because I am more likely to hit on this one.
The first attack against an enemy in a fight is the only one that is independent of other attacks if you are fighting with two Vex weapons.
It is 12.13 the first round, but it goes up every round after that because you do not have an attack that is automatically 100% without advantage dragging you down. Long term as the number of attacks you made approaches infinity the DPR will approach 15 (against a 14 AC). The more rounds you have the closer it will get to 15.
This is sort of false. Yes, you end up with a string, but it is a string that can be broken on any single attack, or by needing to switch targets. Because Vex isn't "advantage on your next attack" it is "advantage on your next attack against the same enemy that you hit with your previous attack" Sure, you can assume you get close to 15 dpr on an infinite scale of attacks, but when you only get 5 attacks against the enemy, that can be a bad assumption.
HM does hurt your damage with pistols and that is the whole point - you are giving up damage by casting/moving Hunter's Mark.
The original post was that it does not cost anything to move/use Hunter's Mark before level 4. It does.
It does not cost anything to move or use Hunter's Mark, in the situation that was described, which was a Dual Wielding Ranger using Nick. That was the situation. Stating "but you can do more damage theoritically by using this other loadout, therefore you are costing yourself THAT damage" is... kind of false.
And also, it should be noted what the practical difference is here. Let us assume for a moment 100% advantage on every attack from Vex, even the first one because the Ranger attacked from hiding or something. Here are the damages.
Dual Pistol - (8.5x0.84) + (5.5×0.0975) =7.676 x 2 = 15.35
Dual Shortsword (vex) - (6.5x0.84) + (3.5x0.0975) = 5.801 x 2 = 11.6
Dual Scimitar + Hunter's Mark (Nick) - (10x0.6) + (7x0.05) = 6.35 x 2 = 12.7
So, assuming Vex equals 100% perfect advantage (which it does not) the Shortswords do LESS damage than Scimitar + Hunter's Mark. The Pistols do still do more damage at this stage, but the difference is 2.65 DPR, which is not a lot.
And, let's say we move to level 8 or so, with maxed feats and attacks, what happens? (dropping shortswords and ignoring modifier increases for quicker math)
Triple Pistol - (8.5x0.84) + (5.5×0.0975) =7.676 x 3 = 23.03
Dual Scimitar + Hunter's Mark (Nick) - (10x0.6) + (7x0.05) = 6.35 x 3 = 19.05
NEXT ROUND being (10x0.6) + (7x0.05) = 6.35 x 4 = 25.4
So, interestingly, on the turn you set or move Hunter's Mark, the Triple Pistol with 100% vex advantage all the time (which won't happen) does more damage, but on a turn where you make 4 attacks against a target you have ALREADY marked, Hunter's Mark does more damage.
And THEN you can take a look at high level, when Hunter's Mark actually does give 100% advantage:
Triple Pistol - (8.5x0.84) + (5.5×0.0975) =7.676 x 3 = 23.03
Dual Scimitar + Hunter's Mark (Nick) - (10x0.84) + (7x0.0975) = 9.083 x 3 = 27.25
NEXT ROUND being (10x0.84) + (7x0.0975) = 9.083 x 4 = 36.33
And at this stage... Hunter's MArk is clearly doing more damage than your dual-pistols EVEN WHEN YOU MOVE IT. While I am granting you 100% perfect advantage on all attacks.
So again, this is a little more complex than you are making it sound. And even when you are correct, the difference is relatively small.
