In order to get an open RPG community off the ground, you need a few things:
1. A central repository that forms the baseline of the rules. This needs to somehow overcome the "Official Source Only" effect to the point where a noticeable number of people look to the repository for their rules. Obviously it can't happen overnight, but it could be a slow growth of awareness. Although it could be sponsored by a publisher, deciding control of the contents should be beyond the sponsor's control. For the rules baseline, the d20 SRD would be a good place to start.
2. An SRD board to review and approve changes. You need decision-making authority at some level to provide consistent, reasonable change control. The board members would listen to community discussion, and there would have to be well-defined criteria for accepting changes, but at the end of the day the board members would be the ones deciding on what changes became official. Again, the board needs to be divorced from any commercial interest.
This doesn't have to be as limiting and constraining as it may sound. Alternative rules could form the basis of forked development streams (consider Grim'n'Gritty vs. Rules-Lite), so that the SRD could support a wide variety of playstiles. The review board would exert some measure of quality control to ensure that rules are well thought-out and contribute meaningfully to the development stream in which they appear.
3. Community buy-in to contribute to this one particular open-source effort rather than go off and play in their own sandbox. To some extent, I think this has been one of the big problems facing the OGL - many contributors wanted to do their own thing with the rules, and few people really wanted to take into account what others had done.
I see the "Official Rules Only" effect as the biggest obstacle. If no one wants to play with the open rules, no one is going to want to spend time on developing new rules for the system.
There's always going to be conflicting desires between releasing rules as open-source vs. keeping them closed-source. However, if the movement begins to grow in popularity, beyond what a small publisher could achieve on their own, publishers will have the incentive to get core mechanics adopted in the SRD to encourage people buying their products that may have interesting adaptations or uses of the mechanics.
Maybe the best motivator for a healthy open rules community is the growth of services-related business opportunities or products derived from rules, but not consisting of rules. That means that adventure products (for example) encourage an open rules system, as the product itself does not consist of the rules, while commercial products that consist of rules (such as a book of feats) actually provide a disincentive to developing open rules.
Furthermore, a D&D service industry, if it's even possible, would provide additional incentive for open rules development. The community of service providers would have it in their best interests to develop the best, free rules set possible, and their distinguishing characteristics would be the type and quality of services provided. I'm picturing a Mini Cooper, with a 3-foot diameter plastic d20 mounted on the roof, and "DM-For-Hire" emblazoned on the side
