I think that rules are important to a successful RPG, but they are only one piece of the puzzle. I suspect that Paizo's bread and butter is really the Adventure Paths, but good adventures and setting material are just one more element of a successful RPG.
We spend a lot of time analyzing what went "wrong" with 4e, and what Paizo did "right", but I really think that the biggest problem has been with splitting the market. This has been hard for Wizards, because it's weakened their sales at a time when their main RPG product is being compared (probably unfairly) to Magic and many unrelated toy products. It's worked well for Paizo, because they've been able to capitalize on the edition wars and their pre-4e strengths to expand and gain respectable sales on what is really just the world's most successful D&D clone. While they have extensively revised the rules, Pathfinder is still fundamentally 3e D&D, and I doubt anyone could prove that the rules are objectively better - perhaps just better tuned to the preferences of people who liked 3e.
We don't have enough evidence to claim what RPGs are actually doing "well" or "poorly", and I'm not sure anyone can even agree what that means. I doubt anyone knows how big the RPG market is at all. However, it seems like 4e and Pathfinder are the top dogs at the moment, and they're both finding different ways to maintain what success they have.
I think the most important thing for a successful RPG, however, is diversified revenue. Wizards noticed that the greatest profitability came from core rulebooks, and rightly or wrongly based their strategy on that. Paizo has always made their money from publishing adventures and optional rules, and their strategy is based on that. But IMO both companies survive because they don't make most of their money from their main revenue stream - Paizo has a successful online store, sells tons of different game aids, and probably sells plenty of material to people who don't play Pathfinder at all (witness the huge number of "4e conversions" of their products floating about). Wizards sells board games, DDI subscriptions, game accessories, and so on, and again, it is likely that many of their customers don't play any version of D&D.
RPGs aren't profitable products, by most any standard, mainly because they are infinitely replayable. It's hard to imagine a less expensive social hobby. Fundamentally, I think that the only way to make money in RPGs is to sell things that aren't RPGs and hope that those other things make you enough money to keep on publishing RPGs.
. A large chunk of that list is the novels and Mark of Heroes/Xendrik Expedition adventures though. If you count just the over-the-counter RPG products, the total for Eberron over four years is only 23 titles; a bit more than the FR had in four years, but still not quite the same level as Golarion. (And if you want to count the MoH/XE material, then we need to also count the Pathfinder Society modules, which will keep Golarion firmly in the lead.)
