Some "random" things...
Magic vs Skill:
If magic can do the same as a skill, you can have different options:
1) Magic is very limited. You can do it with 100 % success, but only once in a short time. That's basically what D&D magic tried to do, but at some point, certain magic was no longer that strictly limited. (either because these spells required spell slots you had in abundance, or you could "magic on a stick")
2) Magic works similar to a skill. Maybe you need to make a magic skill check instead of the normal skill. Invisibility spells in Shadowrun are resisted with Willpower or Intelligence (they are still "better" than regular stealth, but you could avoid that.
3) Magic is not reliable at all. Maybe your invisibility spell can break down or fail when tried to be cast, and retries are hard.
"Spike damage" vs "consistent damage".
That could be your operating model between sneak attack/backstab/inesse fighting styles vs "brute" fighting style.
The Rogue often functions as a scout. He is operating away from the party, and he needs the ability to take down individual enemies fast before he can retreat or move on.
1) Sneak Attacks
This is a unique opportunity that doesn't repeat itself often (or at all) in any scenario. You need to ensure that your enemy is unaware of your presence, sneak onto him, and strike hard and deadly.
A certain assumption we take here: The more agile your weapon is, the better this works.
A Backstab deals 1[W]+DEX+STR damage with two-handed weapons, 2[W]+STR+DEX with one-handed or ranged weapons, and 3[W]+STR+DEX with light weapons, half damage on a miss. If you deal a certain amount of damage (maybe Constitution + 1/2 level), the target must save or be knocked out (dead/unconscious), on a successful save it is just knocked prone, and takes an ongoing effect based on weapon, for example: Light Blade/Heavy Blade/Axe - Ongoing Damage 5 (save ends); Mace/Flail/Hammer - Daze (Save ends).
If we assume the "Martial/Fighter" class basically works like the 3E Fighter (bonus feats), there would be a fighter bonus feat that increases the damage by +1[W] and maybe a few feats that increase the save DC and the bleeding damage/worsens Dazed to Stunned until end of next turn, Dazed (save ends) or Dazed (save ends), Aftereffect: Combat Advantage (save end)
2) Backstab and Brute Fighting:
Backstabbing would be a Rogue typical feat or feat tree. It would increase your damage and/or attack bonus when fighting with combat advantage, with greater benefits for light weapons. Brute Fighting would increase your damage output under any circumstances, but even more with two handed weapons. Add Shield Fighting for those with a shield.
Backstab I: +3 to damage when you have combat advantage.
Backstab II: +1 to attack when you have combat advantage.
Brute Fighting: +1 to damage with one-handed weapon, +2 to damage with two-handed weapons.
Brute Fighting II: -1 to enemy attacks when attacking with a two-handed weapon.
Shield Fighting I: +1 to defense when using shield and one-handed weapon.
Shield Fighting II: +1 to defense of yourself and your adjacent allies with one-handed weaopn.
Or, if you prefer fighters to be more versatile in their combat style:
Weapon Master I: +1 to attacks.
With Light Weapons: Increase to +2 when having combat advantage.
With Shield: Add +1 to defense.
With Two-Handed Weapon: Add +2 to damage.
Weapon Master II: +1 to all damage.
Prerequisite: Weapon Master I
With Light Weapons: Increase to +3 when having combat advantage.
With Shield: Defense Bonus applies to all adjacent allies.
With Two-Handed Weapon: Increase to +2.
Weapon Master III: +1 to all damage.
Prerequisite: Weapon Master II
With Light Weapons: Increase to +3 when having combat advantage.
With Shield: Defense Bonus from WM I increases to +2.
With Two-Handed Weapon: Enemy also takes a -2 penalty to attacks against anyone but you.
The basic idea when balancing the specific bonus:
The Backstaber/Sneak Attacker deals a lot of initial damage, but that "spike" damage (and possibly negation of damage by taking out an entire opponent) is balanced against the extra damage inflicted by someone that needs to rely on consistent damage (and possibly hindering his foes attacks.)
Doing this in a vacuum is pretty hard, you would need to have a good model of your "typical" combat and the expected damages and so on. Well, you need to do that if your goal is actual balance. If you just want different ways to play each character, you can be more imprecise, but don't run to me if people whine about overpowered or underpowered Fighter builds in your system.
----
Skill Systems in general:
I think one "issue" with the system is that you need different number of skills for different roles or sublcasses inside your Fighter class. The Rogue just needs to cover a lot more skills than the standard brute fighter of 3E. If you want to cover both with the same main class, and allow a certain freedom in switching between these roles, you need to have a solution for that.
Not all skills are created equal, not all roles require the same number of skills, and it's hard to put this all under one heading. I am not sure how to solve this. Maybe instead of giving freely distributable skill points (if you keep something like that around), make a certain allotment of skills more or less mandatory (or for free). I personally liked the Iron Heroes skill system with skill groups you had access to. It made it very easy to improve the skills associated with their roles, but if you branched out, it was less effective. It still had its drawbacks..
