I've created 2 games in the past, primarily for personal use. I'm working on a system and setting(s) now that I term something other than a game, but it can be used as an RPG as one of its modes. I'm currently defining my career as a creative designer, so I'm pretty serious about this now.
Here are some thoughts:
1. Learn about what has already been done. Research the most popular and influential games. See how games have evolved over the years. You don't have to know what every game has done, but make sure to look at at least 3 or 4 major contenders for each decade. There are also a plethora of freely available games online. Some downloadable games were even popular commercial games at some point. One edition of Ars Magicka is available as free download, for instance. And of course there are Lite versions of games. Don't limit it to just the big hitters either. Look at some of the smaller Indie games. You shouldn't have to pay a fortune for this. You would be well advised to purchase copies of some of the games you can't get otherwise, but you can get a lot for free. And if you can convince your players to do some of the shopping, even better. It's a great excuse for them to get to try out new systems! What you're going for here is a breadth of understanding of what sorts of systems, ideas, and feels have been done.
2. Determine your vision for your game. Is it about something? I would encourage you to think of the experience you want your participants to have. Is it designed to be Sim<Something> (adventurer, robot, mage, etc), or to be a storytelling game, or to be a Game with loads of dice and cards and balance to rival chess? Is there a genre you are attempting to emulate? How narrow or universal are you going for? Again, focus on the experience, don't worry about the details yet.
3. Once you know what you want people to experience and get out of your game, figure out how best to do it. This is where all that research comes into play. By now, you will have a clear understanding of how detailed or simple a game can be, generic or specifically focused, dice heavy or light (or Other). So from there you can analyze your vision and the systems you've examined, and determine how they would do it. If you think they would do it perfectly...well, at this point you might want to forget about designing that particular game, and just play the one that does it well already. Since you're designing a game, I'm going to assume that none of them will completely do what you want yours to do, and therefore you need:
4. Different stuff. If you are just rearranging what is already there in one or two games, you are making house rules, not a full-fledged game system. Of course, if what you are going for is an experience of a setting or genre--that might be exactly what you are going for--just a new presentation of something people are already familiar with. If that's the case, go for it. Otherwise, make sure you come up with new solutions to the design issues you face. Not gimmicks--solutions. As has been said, don't make something up just to make it up. Make it up because it does it better than what you found, and you need better.
5. Don't be afraid to revise and re-imagine if it improves your product. In some cases, your vision may shift and you might like it that way, and consciously decide to change and go with it. In other case you might have drifted, and you need to catch on to it so you can refocus it back where you need it.
I think that's a good start. Some of the more practical concerns (how often to work on it, how much caffeine you can consume before quality begins to suffer, how can you get friends to forget about meaningless things like MMOs, jobs, or food, and instead spend time playing and critique your system) aren't my forte.
