Sure. And I don't have hard time imagining that humans would treat somewhat intelligent humanoids who they find mildly annoying utterly horribly. It just is that it is the humans who would be bad guys (or at least not good guys) in such a scenario. Such conflicts are perfectly plausible. What I don't get is the need to justify such behaviour by inserting cartoony moral system in it. Just accept that a medieval fantasy world is a a brutal place, your character is not a paragon of virtue, and go stomp some goblins if that's what you want. The desire to paint such behaviour as laudable is what I find distasteful.
Seriously, humans have found reason to fight and kill each other as long as we have existed. No inherent evil and good factions needed, comes to us perfectly naturally. I have no reason to think the same wouldn't be true if sapient non-humans existed.