The d20 System is a complete, fully-developed ruleset. Rolling a d20 to resolve attempts is one rule. From which are you starting? Starting from the former can potentially involve breaking the entire system and then spending large amounts of time trying to fix it, or worse, breaking it and not realizing you've done so until it's too late.I am currently starting from a d20 system, in that the d20 will be the primary form of resolving attempts at things. I don't know exactly how much I'll be changing from there yet. . . Character creation and adding complexity and flexibility to classes is another major point of mine, as I have long been slightly unhappy with what D&D gives me in design space.
@John Out West nailed it, from my point of view, anyway. Start simple and build up. Every new rule you add should either work seamlessly with your existing rules, or operate in a separate space. Here's how I set mine up - the rules are sequential, generally to agree with all previous rules, and they were more or less written that way as well.
Rules Catalog | Modos RPG | Obsidian Portal
Here are the five standard rules modules that make up Modos RPG. These modules are free to use and replicate under a Creative Commons license (read here) when ymodos-rpg.obsidianportal.com
The d20 System is a complete, fully-developed ruleset. Rolling a d20 to resolve attempts is one rule. From which are you starting? Starting from the former can potentially involve breaking the entire system and then spending large amounts of time trying to fix it, or worse, breaking it and not realizing you've done so until it's too late.
If you're looking to roll a d20 to resolve things and add complexity, you can use the rules catalog (above) as your foundation. It's Creative Commons shareware, and designed to be hacked. You might also look at WOIN, if d6s are cool with you. I believe it also has an open license.
My 0.02 SEK, as a long-time gamer (since mid 80's, and having tried a lot of different games throughout the years), and also someone who has proofread around 20 different role-playing products.
1: Make sure you have a clear vision of what it is you want to do.
2: Check if someone else has already made the game you want. You do not want to spend lots of time and effort in recreating something that already exists.
3: Find the USP (Unique Selling Point) of your game.
4: Start small. Make sure you break up the project in small manageable parts. Especially important that you define in advance WHEN you will be finished (not date-wise but scope-wise), and what is good enough quality.
5: Play-test with many different persons. Especially involve people that will try to abuse the system 9 ways to Hell. Take note of how the system handles it. Adapt accordingly. Accept critique here (expect a lot, for things that are broken or ambigously written).
6: Write. Write. Re-Write... Go back to step 5.
7: Have other people proofread it vigorously.. Accept critique, and reflect on their suggestions. Adapt them when needed.
That said. Good luck.
In my case it would be the latter then, I intend to use d20 as the primary form of resolution. I should say that, as I'm most familiar with D&D 5E, that is my starting/comparison point. I won't be rebuilding that entire system in my own way, I just expect that it's inspiration may be obvious in the product since it will be the largest inspiration. A lot of what I have so far, and expect to have going forward, will be my own solutions and systems.
1: I don't quite have a clear vision of where I'm going, so much as I have a few key items I intend to focus on.
2: I don't believe anyone has yet. I intend to have a fairly open and flexible class system, with a few bounds to prevent cherrypicking for a god character. I'd also like a framework for crafting items that I've barely started on, as well as information on how to craft magical items as that is a thing in the world.
3: My USP (I think) will be the class system. I hope to make it flexible enough to build just about any character one could want, from scratch or to imitate a character from media, but still keep it restrained and balanced to prevent being the best at everything. The power sets of the character classes will be a big chunk of the work, but also very modular and adaptable so that I can tweak them individually once I get them to a working point.
I don't know how vague/open I should be this early on in the process, but If people think it would help I could give more specifics about what I'm planning and hope to accomplish.
2: I don't believe anyone has yet. I intend to have a fairly open and flexible class system, with a few bounds to prevent cherrypicking for a god character. I'd also like a framework for crafting items that I've barely started on, as well as information on how to craft magical items as that is a thing in the world.
3: My USP (I think) will be the class system. I hope to make it flexible enough to build just about any character one could want, from scratch or to imitate a character from media, but still keep it restrained and balanced to prevent being the best at everything. The power sets of the character classes will be a big chunk of the work, but also very modular and adaptable so that I can tweak them individually once I get them to a working point.
Classless (and sometimes level-less) D&D-type RPGs is actually one of the common Fantasy Heartbreaker tropes. There is a lot of prior work there to inspire you or perhaps juststeal^H^H^H^H^H adopt. Though if you enjoy futzing with a system as it's own hobby, that may not be for you. (That doesn't describe me. Nope, nope, not at all.)
BTW, I use the term "Fantasy Heartbreaker" in a loving way. I've made them myself, though never played outside me personally running. If you haven't heard the term before it's basically "I want to make D&D - but BETTER". Or "I'm going to FIX all the things I don't like about D&D". In other words, it's aiming for telling the same type of stories with much of the same design aesthetic (combat-focus, ability scores, skill lists, races, etc.) but tailored to what you want to see at the table. They are immensely fun because they scratch your own itch in just the right spot.
But if you want to publish, 5e is already doing quite well in it's niche, and there are plenty of games from blockbusters like PF (and it's new edition) to retro-clones to more narrative d20 games like 13th Age (a Fantasy Heartbreaker that was a recent commercial success) or Blue Rose. So to publish successfully you need to find a different fantasy niche.
But that may be at odds with making the game YOU (and your group hopefully) want to play.
So figure out up front if you want to try to publish and then take a hard turn away from all of those games with their already staked-out market niches. Or if you want to make the game that fits you personally like a glove - but may be too close to an already saturated market area to become more than a "pay what you want" on DriveThruRPG.
This isn't to discourage you. Both paths can be rewarding. It's that they aren't always the same path, and which direction to head is an early decision in design.
So you're trying to invent a 5e-ish, d20 version of Hero System. Or maybe GURPS (Generic Universal Role Play System).
Bear in mind that how vague/open you are will affect the usefulness of replies you get.