If the PCs didn't have family among the squatters (who they'd trust), I'd just get the town so riled up to want to remove them for you. Y'know, frame them for bad things.
Barring that, monsters are the solution.
Steer local wild monsters towards the camp. The wyvern is a good idea, as is say...