I mean, notice how late the Gov's men were to the scene? Minutes after Daryl & Herschel were on scene and had done a sweep, they arrive. While he didn't do it, no reason why that sole vehicle couldn't have been part of a larger force.
Yah, that's what I was thinking. Since the Governor's men show up after the sweep, there's no real telling how many were actually present.