I think you're right. This doesn't make it evil in the "detect evil" kind of way, but it is just looked down upon by people, who'll associate the poison user with the "shady side of town".
Of course it all depends (like you said ) on the circumstances and the kind of poison that is used. I...