Pausing at the 6 hour mark to assess whether to take the high or low road is fine with me.
Thinking it through, if we take the low road, characters who do not have proficiency in Athletics may have an easier time accessing the manor because they will not (I assume) be subject to having to scale a 30' cliff. Another point in favor of the low road = we might be able to take Jote, the cart, and Andal with us all the way to the keep.