Ultimately, I'm not sure that playing even "benevolent" representatives of colonialism sounds like good times. Even the fact that the exiles were literally incapable of going back (even if they wanted to) felt like something to excuse their occupation of dragonborn territory . . . "we have nowhere else to go!" is the colonialist analogue of "but torture's acceptable if it's the only way to save a large number of people!"
I think, for me to be comfortable with the setting, I would need to give the exiles a secluded patch of territory which they wouldn't be taking away from anyone else, but then that lowers the pressure on the exile community to get involved with the dragonborn-tiefling conflict - and what then is the point of the setup at all?
So right now I'm not sure whether it's worth salvaging the Exile idea or trying something else.