One suggestion would be to use a system that lets the players choose how determined they are by declaring what penalties they're willing to take. Essentially, the more penalties they're willing to take for success, the higher the result they're likely to get. I belive Dogs in the Vineyard has such a system.
How I'd run it is something like this:
To determine attack success and failure, roll 3dX where dX is a standard die type from d4 to d12. Whoever has a higher total on their dice wins the roll. Attacker wins, they do damage equal to one half of their roll to their opponent, rounded down. Defender wins, no damage is dealt beyond penalty damage. Ties go to the defender. The base roll uses d4, and you can increase your roll by one step by taking automatic damage where the first two steps require 1 damage per step, and the next two 2 damage per step. Attacker and defender both choose their dice type simultaneously.
So an example of play:
A attacks with a 3d4. He rolls a total of 9 points.
B defends against A, and decides to counter with 3d6. He takes 1 damage, but rolls a 10, so he takes no further damage this round.
B attacks, and decides to use 3d10, so he takes a total of 4 damage automatically. He rolls a total of 15.
A defends, and uses 3d6, so he takes 1 automatic damage, and rolls 10. He has lost, so he takes a total of 8 damage this round.
After this round, A has taken a total of 8 damage this round, and B has taken a total of 5 damage.
Next round:
A attacks with 3d4, taking no damage, and rolls 11.
B defends with 3d4 and rolls 11, and also takes no damage, as defenders win ties.
B attacks with 3d12, taking 6 damage, and rolls 5.
A defends with 3d8, taking 2 damage, and rolls 14, so he does not take any additional damage.
After this round, A has taken 10 damage total, and B has taken a total of 11.
The idea being that this way, you can get closer to ensuring you win against a weaker opponent, but you take more and more self-damage as a result, even though on average, you will only gain a few points of average and maximum damage assuming a weaker or equal opponent, thereby limiting the effect that the larger dice types have over the smaller, less risky types due to the effects of high dice rolls on single dice.
I haven't done the exact probability math to determine the exact amount each die size increase should cost, but I suspect it'd probably be relatively easy to work out and fairly similar. (I don't know how to work around the problem of the d12 though, other than to take it out.)