While I would never put genocide on the table in any of my games, negotiations don't always work and appeasement may not always be the "good" answer. It just depends on the scenario. Appeasement didn't work in the lead up to WWII, it just emboldened the Nazis.
It doesnt matter if it works or not. We don't measure 'morality of a decision' by 'effectiveness'.
For example, sacrificing an innocent child to stop a Demonic incursion (and save the lives of millions) is still an Evil act.
It may be effective. It may be the lesser of two evils. But it's still an evil act.
As a general rule, an Evil person would do it with little qualms. A Neutral person might do it, if there was no other option open, and be remorseful. A Good person almost certainly would not do it and even if they were to, would struggle to live with themselves afterwards.
Then again, D&D is just a game in a world that the DM has complete controls over, it's not the real world. Don't want to deal with "wiping out the kobold race"? Don't make it an option. It's a game, and sometimes people just want to play a game without having to worry about real world ethics.
Genocide isnt just an entire race.
PCs engage in genocide, when hired by the town to 'deal with Hobgoblin tribe living in the woods, that's been causing problems' if they go to the tribe, and slaughter them/ drive them off their lands (unless in self-defense or the defense of others, and no other option is reasonably open to them, and the violence is proportionate to the threat).
Too often I've seen games where the author of a mod (I'm thinking of Living Greyhawk here) tried to set up "moral dilemmas" that were really "No matter what you do, you're f***ed."
That seems to be the fault of the author, and not a problem with moral dilemmas generally.
As you're well aware, I dont like to spring 'gotchas' on players. If a moral dillema is in play (or a question of alignment) Im more than happy to 'break character' and have an OOG chat with the Player about the Gods opinions on the morality of the decision (before the decision is taken) and (if relevant) any impact it might have on the characters
actual alignment (or any game interactions). Ultimately the decision rests with the player; I just ensure as much as possible that it's an informed decision.
Im never going to tell someone how to play their PC. If you want to play a LG person who engages in murder 'for the greater good' then that's a common trope and I'm OK with that. I'll only pause to tell you that (in the eyes of the Gods, and when it comes to your PCs final judgement in the afterlife) he's objectively an Evil person (or at a minimum, committing Evil acts), regardless of how he thinks of himself, the characters own justifications or beliefs or what is written on his character sheet.
If you really want your LG Paladin to ascend to the Seven heavens at Torms side on death, don't slaughter kids. Of course there is nothing wrong with playing a LG Paladin, zealously convinced of the righteousness of his actions as he slaughters infidels. History is full of similar examples.
It's just you're in for a rude shock if you pick up a Talisman of Ultimate Good when it burns the crap out of your hand.