As far as I know, the profits were there. It's just that Games Workshop calculated that they could make more money for the same amount of effort by publishing more models.
Consider this: If Wizards of the Coast only had a limited pool of editors and quality control people for both their Dungeons & Dragons books and their Collectible Card Games, which one do you think they would ditch first?
Well this is the problem, and why we need a happy billionaire to take over the gaming industry who doesn't mind taking a financial hit to publish good games

.
But seriously, RPGs--except for maybe WotC and White Wolf--are really a low-profit labor of love. The great thing about this industry is that the designers and publishers are also fans, at least 99% of the time (this is also why Hasbro owning WotC is potentially problematic). So I think that if anyone hopes they can make an enormous profit off of them, they're playing the lottery. If I was a game designer I'd be ecstatic if I could put food on the table solely through game design, although my guess is that is very, very rare (how many game designers are full-time? Probably only a few dozen--and only a handful with families, and only a few outside of WotC).
So the two main, extreme options for the game designer/publisher seem to be: 1) Either make it part of a larger company that makes its profit elsewhere, or 2) Publish out of your basement when you have the time away from your Real Job, family, etc.
Now most designers and publishers shoot for something in-between, I would say--and it is a possible middle option, just not as realistic in terms of the long-term growth that is required to sustain a business. Actually, the OGL seemed to make it more of a reality for some, but only for time at least--the d20 companies that survived and thrived either cornered a niche or diversified (think Green Ronin, Goodman, etc).
The good thing is that self-publishing continues to improve, so that we have these RPG co-ops online, Lulu, etc. Of course I think PDFs are more of a transitionary technology, kind of like Hybrid cars, and that they'll work much better when you can easily and cheaply print them out as a book with covers, even from home. And it may even be that the Basement Operation will eventually be able to self-publish quality tomes for cheap, or on-demand, so that any old geek can design a game, preview it online, and print it out and ship it when someone orders it.
But the future is so in flux, so it is awfully difficult to ahead more than a year or two.