I'm a bit unsure of that too. Given that the refugees were far enough away at the creek that the Dorinthians at the barricade in the square couldn't defend them in the first place, and so far as I understand, the main orc force is at the square, not off somewhere else, trying to move what's left of Silas' militia, or the penny packet of horse archers with Dartis, back to the square, just leaves the refugees as meat for whatever random orcs come wandering by.