We did too.
Wish I'd done that!
If you do do that, probably best to swap the commanders so Azar Kul is commanding the assault on the city and his deputy is back at the temple. That makes more operational sense for the bad guys anyway, and I think just feels better when you fight Azar Kul in Brindol at the climax of the campaign.