I finally ran stats on my system (roll both attacks, use the better result).
Code:
Hits on A d8 d12 Better result of d8&d6
1 4.5 6.5 5.2
2 4.3 6.2 5.1
3 4.1 5.9 5.0
4 3.8 5.5 4.8
5 3.6 5.2 4.6
6 3.4 4.9 4.4
7 3.2 4.6 4.2
8 2.9 4.2 4.0
9 2.7 3.9 3.8
10 2.5 3.6 3.6
11 2.3 3.3 3.3
12 2.0 2.9 3.0
13 1.8 2.6 2.8
14 1.6 2.3 2.5
15 1.4 2.0 2.2
16 1.1 1.6 1.8
17 0.9 1.3 1.5
18 0.7 1.0 1.1
19 0.5 0.7 0.8
20 0.2 0.3 0.4
Note that the 'better of d8 & d6' option does more reliable damage when you're fighting high AC, but does less damage than two handed when you're fighting low AC. Against 'par' AC, where you need roughly 10 to hit, they do the same damage. That sounds balanced to me.
If you add in 3 from str and 2 from magical bonus...
Code:
Hits on A d8 d12 Better result
1 9.5 11.5 10.2
2 9.3 11.2 10.1
3 9.1 10.9 10.0
4 8.8 10.5 9.8
5 8.6 10.2 9.6
6 8.4 9.9 9.4
7 8.2 9.6 9.2
8 7.9 9.2 9.0
9 7.7 8.9 8.8
10 7.5 8.6 8.6
11 7.3 8.3 8.3
12 7.0 7.9 8.0
13 6.8 7.6 7.8
14 6.6 7.3 7.5
15 6.4 7.0 7.2
16 6.1 6.6 6.8
17 5.9 6.3 6.5
18 5.7 6.0 6.1
19 5.5 5.7 5.8
20 5.2 5.3 5.4
The reason this looks the same but with 5 added to the score is because that's exactly what happens. If you're only ever using the greater result, and never getting the benefit of both dice, it means you're also never getting to add your str and other bonuses twice etc.
The only error I can spot in the above table is leaving out 1.5*str for a two handed weapon. If you include that, two handed weapons are almost automatically going to be better than using dual wield with this system. I considered adding this back in, but then I went searching through my 2e PHB and could not locate that rule. Seems like a 3e-ism. Remember also that in 2&3, a two handed sword was d10, not d12. Considering both of those points, I see no problem with leaving the 1.5 str out when trying to evaluate the three attack systems.
If this system is chosen, there is no need to have feats or specialties to balance out two weapon fighting.
Also, if the 'attacking two targets in the same round' thing is a real selling point, you can rule it as follows:
Nominate one target for your primary attack, and a second target for your secondary attack. Which ever set of dice ends up doing the most damage, that is the only damage you apply this round.
None of the three options (S+B, 2hand, 2weapon) is clearly better than the other in all situations. The two non-shield options are balanced against each other without any attack modifiers, and without any feat cost. How much AC a shield gives would need to be balanced, but doing so can happen without having to alter the other two options. They are in effect, discrete.
p.s. the numbers look even better if two handed weapons use a d10 and don't use 1.5 * str