Imaro
Legend
I think the games that make good use of system mastery tend to have a baseline effectiveness everyone can access and a reward system for mastery that allows more versatility and options rather than vastly higher effectiveness. The way I see it, some systems tell the players to choose "A", "B", or "C" and get locked into one of those choices; where "C" ends up being a bad choice that gimps the character. Whereas systems that do a better job with mastery tend to have a system where everyone can always access all three choices. "A" gives baseline effectiveness in every situation. "B" and "C" are less effective than "A" in some situations, slightly more effective than "A" in others. Experienced players are rewarded for mastery by recognizing situations in which "B" and "C" are better than "A" and choosing to use them "correctly". On the other hand, inexperienced players can just choose "A" all the time without being completely underpowered compared to the other characters.
The consistency and magnitude of the reward for experienced players comes in finding ways to combine options "X", "Y" and "Z" with options "M", "N" and "o" to create a situattion where option "B" becomes more effective than "A" for a little while.
The problem I see here, is that in order for the system mastery to be rewarded... the GM has to make sure those situations where B & C are more effective come up enough times to justify the amount of system mastery it took to realize the difference.
Of course it could also go the other way, where the GM only introduces situations where B & C are sub-par... this just doesn't seem like a viable solution to get someone interested enough to figure it all out.
To your final paragraph... I will say again, the reward for finding these combos has to be great enough to justify the necessary work in figuring it out. Another downside is that if it is group based, like in 4e, the other players may not be interested in trying to figure out, or going along with your plan to combine X, Y & Z.