The best way to make a small fortune in RPGs is to start with a large fortune.
Beyond that...
Traditionally, the biggest selling item in an RPG line (by a long way) was the Core Rulebook (for D&D, the PHB specifically - even the DMG and MM were secondary). So your strategy should really be built to maximise sales of that - indeed, everything else could be a loss leader, provided they led to more sales of that Core Rulebook.
However, that orthodoxy has been challenged recently due to two recent developments, one used by the 'big' companies (WotC and Paizo) and one used by 'little' companies (almost everyone else). (The 'medium' companies still mostly use the traditional model, but a new RPG company should probably avoid this.)
For the 'big' companies, the key is subscriptions, whether those are subscriptions to the DDI (WotC) or subscriptions to physical products (Paizo). The short version is this: if you have 1,000 subscribers, you know ahead of time that you can safely print your next AP volume safe in the knowledge that it will sell enough to be worthwhile. You also know how many copies to print, which again is a huge boon (since printing more is much more efficient, but only if you can be sure to sell them - you don't want to be stuck with masses of unsold books).
For the 'little' companies, the key is Kickstarter. Define your pitch such that you want to fund your Core Rulebook as your main deliverable, have some enhancements to that as your first stretch goals (that is, things like "hardback" and "colour art"), and have the rest of your product line as further stretch goals. Run your project, produce the products you've funded with those stretch goals, and then stop. (Kickstarter probably won't be good for "a second batch of supplements for "Game X" - I doubt such a project would fund, or fund well. So, if you really want to do a full Adventure Path, make sure it's one of the stretch goals for your first Kickstarter.)
Edit: One more thing about KS: Kickstarter appears to have moved from Phase 1 (fad) to Phase 3 (maturity) since last I looked. (Phase 2 is "backlash". I'm not sure if it just skipped that, or if it was just very, very short.) What that means is that while there is still money out there, people are now no longer just throwing their money at anything even vaguely interesting. It also means that you'll find it much easier to fund if you have a 'name' (Monte Cook, Morrus) or you have a track record of success (or, better, both). It's not impossible for someone new to break in, but it's much harder than it was. And if you are doing KS for the first time, and you're lucky enough to fund, then make damn sure you deliver, because if you don't then you won't get a second chance.