Instead of having them be his children, what about having them be several generations removed? Their families went into hiding at first, but now are amongst the general population. Maybe they have family tales that have been passed down. Maybe somebody is searching for them (good or bad). Maybe...