I think it would be pretty bad for it overall. It wouldn't kill the industry, but it would cripple it into being trapped in an ever dwindling niche. The few reasons I see for this are:
1) Maybe culture will change in a generation or two, but for now, unless it's in a professional pretty box on a shelve (even virtually), it won't be widely accepted. Appearances are a big factor in branching out into new customers, and high production standards wouldn't really be possible without good returns.
2) The industry would lose A LOT of great talent. Yeah, there's plenty of great amateur talent, but especially at first, the talent levels would plummet. Look at how many WotC layoffs are doing work part time. A few have tried (aka the Game Mechanics), but most have either been hired by other companies (aka Sean Reynolds at Paizo - but even that was years later) or moved out of the industry. Plus, living and breathing it as a full time job generally leads to richer, better work than doing it nights and weekends when your kids are wondering when you can spend some time with them.
Lastly, I think high-end artistic talent, since that's probably easier to use in other industries, would dry up the worst.
3) The industry standard would probably be along the lines of what's available at RPGNow for $2 and less. Not saying that's good or bad - there's actually both extremes in that area - but the likes of Pathfinder, Tome of Magic, Manual of the Planes, would probably not exist. The vast majority would be small personal projects since large scale coordinated efforts on CREATIVE projects is very hard to manage all the way to completion (I've been part of over a dozen failed ones, and ZERO completed ones). Something along the lines of 4e would probably not happen either because of this.
4) No licensed properties. Star Wars RPG wouldn't exist. Without a clear financial return, I don't see any licensed RPG ever appearing. Of course, there could be fan conversions as long as the copyright holder doesn't notice or care.
5) No printed books unless they were print on demand. But POD is gaining serious ground on traditional printing, so quality is not much of an issue and cost will eventually become more comparable.
However, I think something short of that is still possible. The main open source/shareware business model is actually along the lines of "give away some of the goods, so that they will pay for the rest". In software this is typically either a more robustly featured version or support contracts. Now a big company like WotC could go along these lines with either a limited free version (e.g. heroic tier only, cleric/fighter/rogue/wizard, human/elf/dwarf), and then sell the full version. In fact I think that could really work well, but it's too untraditional for a company as "corporate" in mindset as WotC. The "support" option could go along the lines of giving away the PHB online, but offering DDI subscriptions. That's a little riskier (since the PHB requires A LOT of development costs), however.
Secondly, for smaller companies (e.g. not WotC, White Wolf, or Paizo), I think a shareware model is actually a great idea. Beyond the top companies, it becomes much harder to stand out, and market penetration is a much bigger factor. Being distributed free won't help WotC gain much more market penetration (in their position advertising costs is more bang for the buck than lost book revenue), but for Tommy Publishing that works in dozens, or if really lucky hundreds of sales, free distribution can be a huge boon. In the end, I think it still helps if there is some revenue coming in somehow, but at the small scale if it's even just breaking even, that can help keep someone producing a lot of content much faster than getting absolutely nothing in return. Individual projects may be motivating enough to do it for nothing (which is actually at a loss if you think about what else you could be doing as well as any costs of art, software, website, etc.), but sustained interest often needs more reward than appreciation.
At least in my opinion.