So, out of curiosity, what's the answer? You obviously have a good handle on the math involved, so what is your houserule/patch/fix?
There's a couple of workable answers:
1> Two Weapon Fighting is a feat to increase damage in a Full-Attack Action. In this model you are spending a feat to apply your static damage bonuses twice. From a proportionality standpoint, the single TWF feat should give you an extra attack with every single BAB granted attack you have - otherwise the proportionality is going to turn against you the instant you get a bonus attack and the style is going to suck. In this model you can suffer a penalty to-hit if and only if the expected damage bonuses being doubled are
larger than the penalty involved. So if you just expect the character to have Weapon Specialization (which gets doubled for a total of +4 damage instead of only +2 for a character
without TWF) - the penalty has to be smaller than -2. In practice, if the penalties are not exceeded by the doubled bonus by +3 or more it's simply not worth considering. It wouldn't be "good" unless the bonus was +4 or more.
So using a Full Attack and a feat and suffering -1 to-hit in order to get Weapon Specialization Twice is basically a feat and a full attack for -1 to-hit and +2 damage - which honestly pretty much completely sucks (but at least is pulling ahead). The bonus would have to be more than that before I could consider it to be a worthwhile expenditure of a feat. And remember, if you get
any attacks that you don't get doubled they count backwards. You'd want at least basic damage bonuses of +4 being doubled to justify a -1 to-hit and a full attack and a feat expenditure.
2> Two Weapon Fighting is a feat to increase damage generally. In this model, you apply damage bonuses twice and simply get an attack with the off hand on every single attack you would normally get. That's two attacks of opportunity, two charge attacks, the whole thing.
You are basically running into the same math as above - but you count Attacks of Opportunity and Charges and such as neutral instead of negative. The minimum to make it worth while is a +3 damage bonus being doubled for a -1 penalty to-hit.
3> Two Weapon Fighting is an option to make damage more regular. In this model the goal is to have TWF do exactly as much damage as Two Handed Weapons. That means that characters will expect to pay zero feats for the priviledge. It also means that they should get one attack with each weapon for every single attack they ever get. It also means that the total penalties they suffer should be zero. It also means that any static damage bonuses should apply to only one weapon or be divided equally between weapons.
So in the 3rd edition model, TWF needs to beat Power Attacking by about 3 points (or more) to be worth it, and still needs to apply to iterative attacks and whirlwinding. In a model where you got off-hand attacks with charges and AoOs, it would only need to beat power attacking with a greatsword by 2 points to be worth it.
In a model where characters got off-hand attacks with all of their attacks and suffered no penalties and so on and so forth - it would be balanced at no feat cost if the static bonuses were divided between weapons.
As long as Two Handed Weapons apply double power attack bonuses and TWF don't get static damage bonuses at all - it's hopeless.
-Frank