FrogReaver
As long as i get to be the frog
Count me in the group that thinks this is a useful white-room exercise.
Here is how I think about this problem: There is a combatant fighting an infinite supply of identical enemies, each with H hit points. The probability that the combatant hits an enemy with an attack is p; when the combatant hits, they cause XdN + b damage. We're not interested in how long the combatant will last before going down, so the enemies do not fight back. As soon as the combatant kills an enemy, their next attack can effectively be targeted at the next enemy (i.e., no wasted attacks).
Imagine the "state" of the system is the amount of damage on the current enemy (from 0 to H). Each attack moves the "state" up according to the combatant's probability of hitting and how much damage they cause. We can make a table where the rows correspond to the current state of the system, and the columns correspond to the next state of the system; the ith row and jth column is the probability that the system goes from state i to state j during the current attack. The probabilities of going between states can be encoded to include things like critical hits, whether the attacker chooses to use GWF, etc. As soon as the state reaches H, it starts over again from state 0 (no damage). In this context, "overkill damage" just means that the system can never move beyond state H, even if the damage we've caused indicated we should have (i.e., even we did enough damage to go from 5 to 10, but H = 8, then our new state only goes to 8, so we incurred 2 overkill damage).
The advantage of this setup is that we've actually defined a finite state-space Markov chain, which lets us exploit a lot of powerful math. For example, we can compute the expected amount of time that the system takes to go from state H (the enemy just died) back to state H (the next enemy died), which is the inverse of the rate at which the combatant kills enemies*: if we are considering two combatants with different weapons/fighting styles, we may compare the ratio of their expected amounts of damage (against an enemy with infinite HP) to the ratio of their "kill rate", to get a sense of how much damage is being wasted/how efficiently they are killing enemies.
So, here's how this plays out with some real numbers. Here's our enemy: a 10 HP sack-of-HP that the combatant hits 60% of the time (H = 10, p = 0.6). Our combatant (attacker 1) hits for 1d8 damage (expected damage 4.5). Here's what our damage table looks like (I've included critical hits, and rounded for clarity):
0 1 2 3 4 5 6 7 8 9 10 0 0.400 0.069 0.070 0.070 0.071 0.072 0.073 0.073 0.074 0.006 0.022 1 0.000 0.400 0.069 0.070 0.070 0.071 0.072 0.073 0.073 0.074 0.028 2 0.000 0.000 0.400 0.069 0.070 0.070 0.071 0.072 0.073 0.073 0.102 3 0.000 0.000 0.000 0.400 0.069 0.070 0.070 0.071 0.072 0.073 0.176 4 0.000 0.000 0.000 0.000 0.400 0.069 0.070 0.070 0.071 0.072 0.248 5 0.000 0.000 0.000 0.000 0.000 0.400 0.069 0.070 0.070 0.071 0.320 6 0.000 0.000 0.000 0.000 0.000 0.000 0.400 0.069 0.070 0.070 0.391 7 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.400 0.069 0.070 0.462 8 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.400 0.069 0.531 9 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.400 0.600 10 0.400 0.069 0.070 0.070 0.071 0.072 0.073 0.073 0.074 0.006 0.022
So, for example, attacker 1 misses 40% of the time, and causes 1 damage 6.9% of the time (including critical hits). Attacker 1 kills their current enemy in one attack 2.2% of the time!
The "kill rate" for this matrix is 0.229 enemies per round (remember the expected time per kill is the inverse of this rate). If we consider a new attacker (attacker 2) that causes 2d6 damage, the kill rate becomes 0.323 enemies per round. The ratio of kill rates is 0.708 (attacker one kills opponents 71% times as fast as attacker two). However, the ratio of their expected amounts of damage is 0.643 (4.5 / 7), which is obviously less than 0.708; attacker 2 is less efficient than attacker 1! The difference of these ratios is a measure of efficiency.
There are some obvious special cases. For example, when H = 1, every attack that hits kills an enemy, so both attacker 1 and 2 have identical kill rates (attacker 2 is wasting a lot of damage!). As H goes to infinity, the amount of damage attacker 2 wastes decreases, so the ratio of kill rates converges to the ratio of the expected damage.
There are obviously other ways to approach this problem, but I think this one could be useful!
* these quantities can be computed by solving for the "stationary distribution" of the rate matrix
1. Okay, so I've reviewed your table and can see what you are doing to get each successive row but not why you are doing it nor why it is helpful to do that. It's probably related to the mathematical technique you mentioned which I am unfamiliar with. So care to elaborate a bit?
2. Assuming your calculations and explanations are correct: Why do we care if we are less efficient as long as we are still killing enemies faster?
3. Even if a higher DPR character managed to kill a specific kind of enemy with a certain AC and HP slower than a lower DPR character, Why does that matter in deciding which character or method is generally better than another. I mean isn't it true that there are infinitely more cases where a higher DPR character will kill enemies faster than a lower dpr character (since hp has no actual maximum limit) and outside a few very specific circumstances the higher DPR will start to consistently win out for all HP values above some X and will still win out on a good proportion of those hp values below X.