OK, first post here (after several years of lurking).
I checked out the sales ranking on Amazon.com for several SF games, plus the D&D PH as a benchmark, and the Conan RPG as a popular non-SF licensed game for comparison. Here are the results, ranked from most popular to least ...
D&D Players Handbook 4,448
Serenity RPG 25,186
d20 Modern 37,656
d20 future 49,679
Savage Worlds 68,630
Babylon 5 RPG 2e 76,413
Conan RPG 123,881
Gurps Basic Set: Characters 132,625
World of Darkness 146,049
Star Wars RPG 167,090
Star Trek Roleplaying Game: Player's Guide 238,507
Gurps Space 309,234
Gurps Traveller Interstellar Wars 398,486
Hero System 722,546
So Serenity is currently outselling every other SF game I could think of, including three other licensed games (Babylon 5, Star Wars, and Star Trek) which one might expect to sell to fans who are not normally gamers.
From other posts in this thread it seems Serenity is not selling well in games stores, which suggests that the most of people buying it are not the sort of folks who frequent such stores.
Addressing the OP's question ...
Gundark said:
Is it a good move for a publisher to ditch d20 and develop their own system? And if so, when?
The answer seems to be "yes - when that publisher wants to sell to a fan base which is largely outside the established RPG community." In other words, if they hope to grow the market and attract customers who (perhaps) have hitherto been put off by the complexity of what we think of as the mainstream.
Back to lurking
Nanoc
(edited to put World of Darkness in the correct place)