Since we know the town grew up around the keep, it probably started off this way. I imagine that it was trade, not considerations of siege defence, which determined the direction of expansion. If you have handling of boat cargo south of the cliffs, it's not long before you start building...