MrHotter
Explorer
How about as a house rule allowing re-rolls for any result of 3 or less?
Using a longsword one-handed with Dueling gives a 6.5 average damage. Using it two-handed and re-rolling 3 or less would give you an average damage of 6.55.
For a greatsword this would increase the average damage from 7 to 8.5. For a greataxe it would increase it from 6.5 to 7.625.
That's a lot of re-rolling when you have multiple attacks with 2d6 damage.
You can do what you want at your table, but trying to balance style damage based on using a longsword in two hands for great weapon fighting style sounds like a bad idea for trying to balance the weapon styles.
If you are the DM and your great weapon person is complaining about his damage compared to the dueling style guy, you should just smack him and then add a dueling master feat with a -5/+10 damage (plus some other advantage).