It was canceled after half a season. By definition, it's a failure.
As for quality, that's totally subjective. In my opinion, Firefly was good - but that's all. Not great, and there was nothing particularly special about it. The writing and acting were above par, and the stories were good once they got the initial intros done (it had one of the worst series openers I can ever remember seeing).
Character-wise, they got one thing right - Captain Malcolm Reynolds. Everyone else was severely lacking in depth. The doctor's sister, for instance, only became a remotely interesting character in the last episode that aired - and since a disproportionate amount of the plot revolved around her....