My suggested change, if any, to the -5 attack / +10 damage options in both feats is to make it scale equal to proficiency bonus. Start at -2 / +4 and scale up at a rate of -proficiency / +2 x proficiency. It makes it actually usable at lower levels with the player overshadowing all others in damage, and "buffs" so the player doesn't get upset their feat got "nerfed". I say "buffed" because most games don't reach level 17 anyway.
My larger issue with those two feats has never been with the damage bit anyway.
...
Almost everyone else who think SS/GWM are overpowered specifically talks about the damage as part of it, so I think that's still worth addressing. I'm also going to address it without your house rule changes to the Archery fighting style so that it's generally applicable.
The hit penalty/damage bonus comes into play when the normal expected damage per attack ( hit% * expected damage on hit ) is less than after adjustment ( (hit% - X) * (expected damage on hit * Y)). Crits change this a little, but for the most part that math favors it's use. I'm going to leave it out for simplicity. This is also just talking about the math, not the situation. If a foe has 3 HPs left, you wouldn't use it. If it goes next and the damage bonus will likely kill it, you likely use it to deny the action. That sort of thing is outside the math below, we all realize there are exceptions. Also not discussed are things like sneak attack that only apply once per round and multiple chances to hit in a round, just to keep things easy.
SO, let's look the the supposed 50/50 bounded accuracy change to hit that will never, ever happen if someone is optimizing toward SS/GWM (and therefore optimizing to hit).
50% * damage = D/2 (a/k/a) 2D/4
25% * (damage + 10) = (D+10)/4
The balance point is:
2D/4 = (D+10)/4 ... 2D = D+10 ... D=10
If you average 10 points on a hit, with a 50/50 shot it doesn't matter if you use the feat. That's not hard to reach. A 1st level greatsword with a +3 str is at that. At low stats it's harder to reach with a longbow or hand crossbow, making SS a damage boost even at low levels.
But that's a theoretical point, assuming you aren't trying to get a bonus to hit. Character to hit, if that's something you focus on, can increase a lot quicker than AC. Increasing attack stat, taking the archery fighting style, having Bless or other buff up, finding magic items, debuffs on foes providing advantage - when you take that into consideration the chance to hit can skyrocket.
So, if we accept that to hit can grow faster than AC, then 1st level is the "worst" for these feats and they only get more powerful. So let's look at a 1st level character and assume these feats get more powerful from here.
To have SS at 1st we're talking a human variant, so let's go 16 dex after racial. Add in archery fighting style and you have +7 to hit (+2 prof, +3 dex, +2 style). expected damage is d8+4=8.5 on average. Let's assume at 1st that there aren't enough spell slots for Bless or advanatage through debuff to be standard.
Against a "standard" CR 1 opponent we're looking AC 14. A 70% chance to hit.
So let's do some math.
Baseline = 70% * 8.5 avg damage = 5.95 expected per attack.
-5/+10 = 45% * 18.5 avg damage = 8.325 expected per attack (+40% damage)
-prof/+2xprof = 60% * 12.5 avg damage = 7.5 expected per attack (+26% damage)
Looks like -prof adds less, eh? That's true if all you fight is average AC monsters which is a good start toward balancing. But we also need to look at the range it still is worthwhile.
AC 18 is pretty dang good for CR 1 foes. that's back to the 50/50% with a +7 to hit.
Baseline = 50% * 8.5 = 4.25
Base feat = 25% * 18.5 = 4.625
-prof = 40% * 12.5 = 5
At these higher ACs, the -prof/+2xprof is better than the book feat. But neither has a big bonus.
What I draw from this is:
At 1st level, with foes doing a bell curve around average AC, SS will give a good bonus to damage for a character trying to optimize it's use, +40%/+25% against average AC opponents. For lower AC foes the base SS will have a greater increase but at higher ACs the -prof house rules SS will eventually overtake it, including getting to a point where -5/+10 still isn't useful but -2/+4 still is. This means that outside on situational needs like low HP target or disadvantage, the -prof version should be "always on" because you won't reach an AC where it's a penalty.
Pulling back in that modifier to hit increase faster than ACs, as levels increase SS will get more powerful. The base feat and the houseruled feat will eventually converge at -5/+10.
EDIT: I just realized I messed up the base damage, it should be d8+3=7.5 average. This makes the feat even more powerful. I don't feel like redoing all the math though, this took some time to calculate.