The cylons began as machines, as just robots. Hence the epithet "toasters" to describe them. However, now Cylons are much more advanced. They are apparently using cloned and genetically engineered bodies that are so close to human that a detailed medical examination cannot tell the difference, only a very specialized test or exposure to high levels of radiation for hours. Sleeper agents can even think they are human, and pass for being human for years in the human population. They have their own religion, their own culture. They can even sexually reproduce with humans.
That's not just a "machine", when members of the fleet start to wonder if they could be sleeper agents and not know it, your enemy is not just a machine.
It's one of the oldest tactics in warfare, to dehumanize your enemy, to make them seem less human or less civilized than you, so you can feel a lot better about killing them. It's been going on for millennia, and it still goes on today, and it goes on in BSG. It's easier because the original cylons were inhuman, they were just renegade machines out to destroy all humans. Now they've become something more, something beyond normal limits of humanity.
Helo was having a moment of actual morality, it's been a recurring theme of the show, since the miniseries, of what is the real virtue of mankind. As Adama asked at Galactica's decommissioning ceremony, what had humanity done to deserve to exist? If mankind resorts to genocide, how are they any better than the Cylons? Roslin is quite moral. . .with regards to humanity, but she also sees the cylons as inhuman and has no qualms about torturing or throwing them out an airlock. Adama has gained some degree of insight from the year spent above New Caprica with Athena, hence his reluctance to go ahead with the plan, and being willing to drop the matter when Helo apparently makes a stand (perhaps realizing that in his place he may well would have done much the same).
Individual cylons apparently do have some degree of free will, hence Athena, seen as a traitor by her kind, and Caprica-Six, a huge walking scandal of a human-sympathizer among her kind. New Caprica was a huge failure as an attempt at peace, because the Cylons understand humanity very poorly, just like apparently the humans understand the Cylons very poorly.