You do have a frequency of kill factor. The point is that you are applying it as if only the PC in question could be the one killing the enemy. In the actual game state that doesn't happen.
1. The benefits of doing less overkill only applies to enemies your PC actually kills - we agree there and you listed a factor for it.
2. Your PC will not be the one killing every enemy in an encounter. (Rule of thumb might be to estimate that he kills 1/4 of the enemies in the encounter).
You did not account for #2. Your calculation had the defacto assumption that your PC was killing every enemy encountered.
In general, it's never guaranteed that the 1 v N case can be substituted for the M v N case and yield correct results.
Again, you do not ever care about overkill damage UNLESS the PC in question lands the killing blow. It does not matter what every other case is, it only matters when and if the given PC lands a killing blow. I did account for #2 in that I don't care about situations where the PC doesn't land the killing blow. This is covering in Assumption 1 -- we only care about overkill when a killing blow lands. After that, it's a matter of determining what frequency is appropriate, and here there is room for debate. If, as you say, a PC is assumed to land the killing blow on 1/4 of the enemies presented, and we assume 3 rounds of combat, that's 1.25 kills per combat per PC. If the PC in question has 2 attacks, that's 6 attacks total. If we eyeball it, that's an f of 6, which I happened to include in my example.
Now, I disagree that f should be 6, as a high average attack damage character has a wider kill range and so will more often land killing blows.
Fundamentally, what my analysis shows is that the overkill effect is dependent on two factors -- base average damage and frequency of killing blows. If is directly proportional to both -- as average damage increases, overkill effect increases and as frequency of killing blows increases, overkill effect increases. Nothing in this analysis says that the reference PC is doing all of the killing -- the rate of killing blows is an input variable, for goodness sakes!
If we take your assumptions above, and look at the previous example of rogue versus monk, we can do a rough analysis. Both have a +4 in the controlling stat, and both are 5th level. That's +8 to attack and +4 damage from stat with the Rogue at +3d6 sneak attack damage and using a +1 sword in the main hand. Let's assume the target is AC 14, so hit% is .75.
dmg(R1) = 1d6(weapon)+3d6(sneak)+4(stat)+1(magic) = 19
dmg(R2) = 1d6(offhand weapon) = 3.5
dmg(Rperround) = 22.5
dmg(M) = 1d6+4 = 7.5
dmg(Mperround) = 3*dmg(M) = 22.5
dmg(Rperround) = dmg(Mperround)
OAD(R1) = 6.625
OAD(R2) = 0.8125
OAD(M) = 2.3215
If we set frequency of killing blows to 1/4 over three rounds, the rouge makes 3 main hand attacks and 3 offhand attacks and the monk makes 9 attacks. The rouge has 2 cases now, one where the mainhand is the killing blow and one where the offhand is the killing blow. Let's assume this breaks according to the strength of the attack or that mainhand is going to be killing blow 19/22.5 times. Offhand the remaining 3.5/22.5 times. This is very close to .85 and .15. We'll adjust this when we solve.
This complicates the rogue, but we can do it using this split and calculating X' independently for each attack. The necessary step is in the calculation of X'. You have to multiply the OAD in that calculation by the split, so the new X' for the rogue looks like:
X'(R1) = X(R1) - ((.85*OAD(R1))/3) = 12.37
The frequency of 3 is used because the rogue makes 3 of these attacks. The percentage modifier adjusts this to reflect the times that one of these three attacks is the killing blow. X'(R2) is solved for the same way:
X'(R2) = X(R2) - ((.15*OAD(R2))/3) = 2.61
X'(Rperround) = 14.99
The monk is straightforward.
X'(M) = 5.37
X'(Mperround) = 16.11
As you can see the difference in effective DPR when accounting for overkill is slightly in the monk's favor by 1 DPR.