Thanks everyone for the comments and suggestions. I am currently first looking at Core Mechanics, but am interested in other system aspects as well; any favorite mechanic really, though Combat systems and Magic systems are pretty high on the priority list. I am doing research because my group is currently messing around with making a system of their own and I'm not real keen on where they are going with the mechanic.
Currently the concept they are going with is the core mechanic will be based on a single roll of XdY, the single highest die result is your result, but if you get more than 1 of the highest die result your result goes up by +1. (ex: if you have 2d6 and roll 2 6's the result is 7)
X and Y are determined by Attribute Rating (1-5: determines number of dice) and Skill Rating (d3 - d12). So if you have say Agility Attribute at 3 and Stealth skill Rating at d6, your roll for Stealth would be 3d6, opposed by say a guard's observation rating.
They are currently looking at using the 9 nWoD Attributes and then each skill being tied to an attribute. They also plan for Most rolls to be opposed rather than static target numbers wherever possible, for instance even a Lock would have an die rating and get to "roll" its resistance to being picked (ex: a simple lock might have a 2d4 rating, so as long as the PC can beat the GM's possible max roll of 4, they'll succeed).
There are also some other rules such as:
* Max of 5 dice regardless of bonus dice.
* Step Up: Step up your die type by 1 at the cost of 2:1 dice ratio (round down). Ex: 3d8 steps up to 1d10
* Step Down: Reverse of step up, but still at 2:1 ration. Ex: 3d8 steps down to 6d6 (technically 5d6) as they keep refusing to look at an overflow mechanic for dice beyond 5.
I've pointed out flaws in the math, but currently falling on deaf ears.
That said...
We've looked at many other games and they are stealing concepts from various systems such as being a lifepath based system similar to Burning Wheel, using concepts like Aspects from Fate or Traits form Numenera, etc. However, for my own sanity, I'm toying with doing my own spin-off with what I consider a more balanced and affective Core Mechanic.
Oh, and as a quick comment to Saelorn...
I'm not sure exactly what you mean by same probability distribution for d20 and 3d6 systems. d20 is Linear and 3d6 a Bell. Very different probability curves. For instance in d20 you have deviation of 5.77 from mean with a flat 5% chance of any number, but in 3d6 you have only a 2.96 deviation. It may not seem much different but it makes quite a bit of difference. For instance you have a 5% chance to roll a 10 in d20 but a 12.5% chance of rolling a ten in 3d6.
Currently the concept they are going with is the core mechanic will be based on a single roll of XdY, the single highest die result is your result, but if you get more than 1 of the highest die result your result goes up by +1. (ex: if you have 2d6 and roll 2 6's the result is 7)
X and Y are determined by Attribute Rating (1-5: determines number of dice) and Skill Rating (d3 - d12). So if you have say Agility Attribute at 3 and Stealth skill Rating at d6, your roll for Stealth would be 3d6, opposed by say a guard's observation rating.
They are currently looking at using the 9 nWoD Attributes and then each skill being tied to an attribute. They also plan for Most rolls to be opposed rather than static target numbers wherever possible, for instance even a Lock would have an die rating and get to "roll" its resistance to being picked (ex: a simple lock might have a 2d4 rating, so as long as the PC can beat the GM's possible max roll of 4, they'll succeed).
There are also some other rules such as:
* Max of 5 dice regardless of bonus dice.
* Step Up: Step up your die type by 1 at the cost of 2:1 dice ratio (round down). Ex: 3d8 steps up to 1d10
* Step Down: Reverse of step up, but still at 2:1 ration. Ex: 3d8 steps down to 6d6 (technically 5d6) as they keep refusing to look at an overflow mechanic for dice beyond 5.
I've pointed out flaws in the math, but currently falling on deaf ears.

That said...
We've looked at many other games and they are stealing concepts from various systems such as being a lifepath based system similar to Burning Wheel, using concepts like Aspects from Fate or Traits form Numenera, etc. However, for my own sanity, I'm toying with doing my own spin-off with what I consider a more balanced and affective Core Mechanic.
Oh, and as a quick comment to Saelorn...
I'm not sure exactly what you mean by same probability distribution for d20 and 3d6 systems. d20 is Linear and 3d6 a Bell. Very different probability curves. For instance in d20 you have deviation of 5.77 from mean with a flat 5% chance of any number, but in 3d6 you have only a 2.96 deviation. It may not seem much different but it makes quite a bit of difference. For instance you have a 5% chance to roll a 10 in d20 but a 12.5% chance of rolling a ten in 3d6.
