That's not accurate. The town, while not as major as the elves, is still an established fact of the setting. It would be easier to change the town than the elves, but still significantly harder than making an encounter 4 elves instead of 6.
Also, the town is in flux and cannot be exactly as...