The reason FOX hated Firefly, in Whedon's own words, was that it was a show about a bunch of nobodies living on the fringe and hiding from authorities. Fox executives like shows that cater to fantasies about being a major player--a part of the "it" crowd. That can range anywhere from a bunch of spoiled-brat rich teenagers to a tireless government agent who's an unstoppable kicker of terrorist butt. This is why Whedon thinks Dollhouse will have a better shot.
As to why shows like BSG, Andromeda, and Stargate keep going forever while shows like Firefly and Farscape fail, it seems like it's largely a result of conditioning. Sci-fi fans want their science-fiction nice and regimented, with established chains of command and unflappable leaders making inspiring speeches where they draw the line against the bad guys. Having a bunch of misfits constantly bickering and infighting and downbeat situations is apparently a turn-off.
As to why the movie failed, it's mostly the title. In order to profitable, the movie needed to attract more moviegoers than just the fans of the TV show. Considering that, calling the movie "Serenity" was a flat-out stupid move. Imagine all the folks going to the theater, scanning the marquee, and seeing a movie called Serenity. The irony is kind of lost.