The balance problem with "SA with spells" is honestly because of Cantrips.
A rogue who attacks with a scaling cantrip does more damage than one who does not. And ideally you don't want every rogue to have to be a spellcaster to be effective at doing damage.
This was already somewhat broken by SCAG weapon cantrips; they scale like cantrips, and are compatible with rogue SA.
Burning spellcaster levels to get a spell, like shadowblade, isn't as bad as cantrips. Cantrips require a feat or a single level dip, and scale up to level 17.
As it stands, the existence of Booming Blade and Greenflame Blade means that non-gish rogues damage output requires a lot of effort just to keep up. But it is limited to melee, and dual-wielding weapons can help compensate (two attacks either gives 2 chances to land a sneak attack, or more chances to crit). It still requires a lot of effort; for example, Champion 5/Rogue 15 with two short swords and a source of advantage (!) can match a AT 20 rogue using something like shadowblade and/or booming blade (or at least be in the same league).
At range, no scaling cantrips work.
Now, the damage output of Cantrip+SA isn't that broken. So you could just grant something like it to all rogues that is incompatible. Like...
Called Shot: Starting at level 5, as an action you can aim for a weak spot. You make a weapon attack (either melee or ranged) and deal an extra set of weapon damage dice if it hits. Starting at level 11 it deals 2 extra set of weapon dice, and at level 17 3 extra set. Only add your attribute bonus once. If you are wielding 2 light melee weapons, afterwards as bonus action you can also make a called shot with the other weapon.
Your DM may offer to exchange your extra damage for causing an effect, like wounding an arm holding an ally of yours to get the creature to let go.