Well, you could always leave those followers back home. It would require a HQ of sorts but I've always thought that most of them are commoners and experts that are willing to join your cause and advocate it to the best of their ability. Also, they wouldn't be in need of any special protection...