Well, if you're a slaver that makes it pretty easy.
<Jersey accent>
It's all business, you know?
</Jersey>
It's all about the black market. You probably don't supply much to civilized folks in Talingarde, but there'll be a market. Demihumans like orcs, goblins and the like love slaves. You could have run a legitimate 'side' business, perhaps hunting bounties. You capture someone, take a trophy as proof of a 'kill,' and then sell the unfortunate as a slave to pad out the reward.
Or you could have gone all in, perhaps even coordinating slave raids on outlying settlements and homesteads; whole families just vanishing in the winter months when help is days away.
Either way, the ruling body of Talingarde was a problem. Limiting your market, making your life difficult and dangerous. You knew they'd come after you...why not go after them first?
We could even have a link, you and I. My character was stirring up trouble...trying to fashion a revolt. Your ill-gotten gains could have been helping bankroll the movement. Until it all went wrong.