What's the main "hook" of the Traveller system? Is it that it specifically ties into the whole "space as exploration" genre?
What's its most compelling positive trait relative to other systems?
Paizo appears to have all of the Mongoose Traveller series at cover price (core rulebook hardcover around $40), so getting my hands on it doesn't appear to be a problem.
What would be the biggest strengths and weaknesses of choosing Traveller vs Savage Worlds for a Serenity campaign?
Traveller has spaceships.
Seriously, that's a major thing to consider. I don't know much about Savage Worlds. It's on my list of things to try, but I've just never had the money and nobody I know owns a copy. Are there good rules in Savage Worlds for spaceships?
Someone else in the thread mentioned Gurps. If you like Gurps, they do have a number of excellent space related books. I certainly have a number of them, but I mostly use them for ideas to supplement my Traveller collection.
So what makes Traveller a good choice? The default theme of the game is very similar to Firefly. A small crew of independent freelancers, hoping from world to world looking for opportunities. Trade, piracy, mercenary work, all of these are common Traveller adventures. Mongoose Traveller introduced a system of "Ship Shares", where starting characters basically get a downpayment on a ship. They are expected to pool them together so each of them have a stake in the vessel that is purchased. Usually it is hard to start out owning a ship outright, so they may own 75% of the ship and have to work to make payments on it.
So the default theme of Traveller fits well with Firefly. The default setting is a much larger space, the Imperium is millions of worlds spread across the galaxy. If you want to move it to a tighter, one system setting like Firefly you just need to remove warp drive. So instead of taking a 1 week to get 1 parsec, make it take a 1 week to get 1 AU.
One of the parts of Traveller's setting that really works for the quasi-legal freelancers is the speed of communication. In Traveller the default assumption is that information is slower than ships. In order for news of a raid, or a wanted poster to arrive at a planet someone has to fly there and deliver the message. So there can be a race to unload stolen goods at port before news of the theft arrives. I think faster than light travel with light speed communications adds to the old west in space feel of the game. It is one thing about the Firefly setting that actually works against it. In a single solar system radio communications will out-pace ship travel if you don't have faster than light travel. Still, all the tools are there in Traveller to make a Firefly type game work.