I really don't think that in the case of Firefly it couldn't have found an audience. Really the only mainstream barrier was getting over the "in space" element. I know from personal experience that it can be entertaining for folks who "don't like sci-fi," just as shows like Heroes, Battlestar Galactica, and Lost can gain a mainstream audience.
What killed Firefly was the network, pure and simple. The pilot episode explained why the relatively large ensemble cast was all together in the first place, and set the basis for the plot arc of the season. Fox, in their infinite wisdom, did not air the pilot, fearing it was too much exposition and not enough action. (Interestingly, they seem to repeatedly misunderstand that what Whedon is most loved for is memorable characters and dialogue.) They then proceeded to pre-empt it and show it out of order, ignoring the fact that it was show that benefited from watching it in sequence.
The advertising, IIRC, was anemic at best for a new show. They clearly weren't sure what to do with it from a marketing perspective. It was undoubtedly pigeon-holed as an odd hybrid "sci-fi western" rather than as an ensemble action-drama a la Flash Forward -- had it been marketed as such and broadcast regularly, it might have found an audience.
Luckily for fans, the DVD sales were sufficient to convince a studio that a major motion piction was worthwhile, and so Serenity was born. Ignore the warnings against it if you're used to Joss Whedon's other work, like Buffy the Vampire Slayer, as you're probably also used to him sometimes twisting a knife into your gut, plot-wise. It's a very entertaining movie, and a must-see for any fan.
There are also comics written by Whedon that fill in some of the story between the series and the movie. If you're lucky you can find it at a Big Box bookseller or gaming store that stocks graphic novels.
