OK, first post here (after several years of lurking).
I checked out the sales ranking on Amazon.com for several SF games, plus the D&D PH as a benchmark, and the Conan RPG as a popular non-SF licensed game for comparison. Here are the results, ranked from most popular to least ...
D&D Players Handbook	4,448
Serenity RPG	25,186
d20 Modern	37,656
d20 future	49,679
Savage Worlds	68,630
Babylon 5 RPG 2e	76,413
Conan RPG	123,881
Gurps Basic Set: Characters	132,625
World of Darkness	146,049
Star Wars RPG	167,090
Star Trek Roleplaying Game: Player's Guide	238,507
Gurps Space	309,234
Gurps Traveller Interstellar Wars	398,486
Hero System	722,546
So Serenity is currently outselling every other SF game I could think of, including three other licensed games (Babylon 5, Star Wars, and Star Trek) which one might expect to sell to fans who are not normally gamers. 
From other posts in this thread it seems Serenity is not selling well in games stores, which suggests that the most of people buying it are not the sort of folks who frequent such stores.
Addressing the OP's question ...
	
		
			
				Gundark said:
			
		
	
	
		
		
			Is it a good move for a publisher to ditch d20 and develop their own system? And if so, when?
		
		
	 
The answer seems to be "yes - when that publisher wants to sell to a fan base which is largely outside the established RPG community." In other words, if they hope to grow the market and attract customers who (perhaps) have hitherto been put off by the complexity of what we think of as the mainstream.
Back to lurking
Nanoc
(edited to put World of Darkness in the correct place)