BSG has SFC, and Firefly did not.
For all that I've questioned SFC, and don't watch the channel anymore, that was the advantage. If Firefly aired on SFC, it may have had a lower budget (though BSG did alright), but I've little doubt it would have aired longer.
To be a little controversial, I think Whedon was a little overconfident if he thought the numbers Buffy/Angel were bringing in (~4m viewers) would float on any major network. Shows pulling in those numbers are routinely canceled. I suppose he was hoping for a big hit on a major network, and felt Firefly was it.
Now, going back to Fox for Dollhouse was flat out crazy, IMO.