I would really like a versatile fighting style, but I haven't come up with a solution yet. I have come up with some parameters to use to help come up with a solution though.
First, I’ll start with some assumptions:
1) Don’t worry about feats. Yes, there are issues, and there are plenty of different house rules depending how you want to fix it (nerf the strong ones vs. buff the weak ones, etc). I’m going assume that you can take care of that balance completely within the feats themselves, so they aren’t relevant for fighting styles themselves.
2) Math needs to fit. Let’s run some numbers to have a basis of comparison with other fighting styles.
With Dueling style you have 6.5 damage and +2 AC.
With Great Weapon style you have either 7.3 damage, or 8.3 damage (depending on whether you are wielding a d12 or a 2d6 weapon).
So we know we need to keep damage below 7.3, and probably shouldn’t give better AC than the +1 with a shield you are using with Dueling style.
It’s also important that we don’t inadvertently make a stacking problem with other fighting styles. For instance, if you could stake the +2 from Dueling with versatile damage, you’d end up with 7.5 – stronger than a greataxe with Great Weapon style.
3) Characters aren’t usually going to switch their armaments. You aren’t going to put on or take of a shield in a battle, and you really aren’t going to be considering each round whether a +1 AC or +1 damage is ideal. So we can’t have those as the primary meat of the features.
Personally, the versatile discussion for me is less about being able to use a shield or not, and more about character style. I want to be able to forgo a shield, and make attacks alternatively one or two handed—not be stuck with using my versatile weapon two-handed as an alternate Great Weapon style.
In that light, I’ll suggest a couple of frameworks (not specific numbers/effects) that might work.
A) There should probably be a passive benefit for the style. This applies regardless of how your are wielding the weapon, and depending on it’s nature might even work with a shield.
B) There should be an active feature that requires you to attack with two hands. Note: this isn’t about deciding whether you are wielding your weapon one or two handed—we don’t want that to be a thing. Rather it just means that you have to empty your other hand for the attack. (This means you are operating the same as a normal two-handed weapon wielder is for these purposes: you can hold a two-handed weapon in one hand, you just can’t make an attack with it that way, so you would have to drop whatever you were holding/grappling, etc, if you wanted to attack.)
C) This style should be competitive with the other styles, but should appeal to those who want to fight with a versatile weapon, sometimes striking with one hand and sometimes with another, rather than those who want to fight another way. Sword and board fighters shouldn’t be taking this instead of Dueling or Protection because it seems better, neither should greataxe fighters be taking this instead of Great Weapon style (though it might be close there, since d12 weapons gain significantly less than 2d6 weapons from the style).
I’ll let others see if they can think of something, since I don’t really have anything satisfying that’s occurred to me yet. Having two different benefits is already more complex than existing fighting styles, so we want to simplify it as much as possible while avoiding any interaction issues.