(I just wrote a blog piece on this very subject...)
I think it's easy to see why RPGs aren't more popular, and that's because there's significant "buy in" to get an RPG group up and running, and most people aren't willing to make that "down payment" in terms of time, logistics, and resources. In actuality, there just aren't enough "true gamers" to support our hobby on the wide-scale level that video games and their ilk enjoy. There are plenty of "casual gamers"; i.e., people who don't mind showing up (semi) regularly, rolling some dice, knocking back some sodas/beer, and heading home, but those gamers aren't the lifeblood of the hobby and they're not generally expanding it. Because of time, family, or life commitments they play if the "table is set for them" - but they're not the types to really read or master rules outside game night, religiously buy the books, or truly delve into our hobby beyond whatever immediate game that's been introduced to them.
All of the above often falls to the "true gamer" - and in most game groups, there is only one, and that's usually the DM/GM. Thus, even in my group - which meets every weekend and has six regulars - if I want the other players to pick up a new game, I have to do all the work. Buy the book(s), make the characters, teach the mechanics or create cheat sheets for combat, etc. One of the great legacies of the D20 era was that once you mastered the mechanics, you could pick up a wide variety of games implementing that same "engine". Now, it seems each new game has a set of new (often even more stratified) mechanics, and although for us "true gamers", that's not a problem (we're going to sit down and read them anyway, no matter how complex, because it's what we enjoy), for the people we're trying to broadly introduce our hobby too - that's a huge problem.
Thus, I would urge any company producing a new RPG to keep this in mind: make the game easy for me to teach and present, then I can teach it to that many more people. Here are some simple things that I think work:
- Premade PCs: Make a set for me, so I can tear 'em out and hand them out. And make them self-contained: i.e., everything my player needs to run that character is right there are on the sheet. Because usually only one person has the gamebook, and if I'm running a new game, usually I'm the one who needs it...
- Combat Cheats/Tear Sheets: Prepare them for me so I can hand them out to the players. Between their character sheets and these tear sheets, that should be all the new players need to dive in and actually start playing...
- Intro Module: Make a real intro module, complete with all the monster/creature/antagonist stats and everything I need to run it "right out of the box", and make it so it highlights the features of this game...
- Easy Rules: And I know this is "easy" to say but harder to quantify, but I sometimes think game companies make their games more complex for creditability rather than playability. Complex rules are great for true gamers, because part of the reason we like these games is because we enjoy dissecting, exploiting, or rewriting the very rules themselves, but for casual gamers, it's a problem. They don't care about the rules - they just want to play...
With these things in hand, I can throw a new game - even with new rules - at my group without much angst. With premade characters, I can start a game without wasting a whole game night making PCs (and if the group likes the game, they will then eagerly invest the time into making characters later). With combat sheets, I can get into the game without wasting hours explaining the rules, and with a premade adventure, I can highlight what is best about the game and why we should run a campaign in it, without spending hours having to do so by myself before we even sit down to give it a try.
And what I've found is that if I can introduce players to a new RPG in a quick, fluid, and "easy" way, many times I can turn those "casual gamers" into "true gamers", at least for that game...
...until I can spring another on 'em...
I get a few hits 
