There's a lot of this adventure that doesn't make sense. "Follow the treasure" is the mission. That's what they are doing. But since no actual speed is given for the castle, only "The wyverns can catch up because the castle is slow", it makes sense to "follow the treasure" not suddenly turn...