First of all, Followers are not Fanatics. For that matter, Cohorts usually aren't either. But, there is no reason why a player shouldn't have to work to recruit all of those followers. You don't take the feat and they appear on your doorstep the next day.
I tend to see followers as loyal serfs/yeoman in a feudal society. They make a good basis for a village or a town. They can be the backbone of the town so that everyone keeps working toward the good of the Lord (The PC). New people may move into the town and it grows in prosperity, but the new residents won't have the same loyalty. Over time, some followers will die, others will fill their place. Maybe the local smith dies in an accident. The PC now has a free follower slot. But, if the PC want the smith's son to take his father's place, including the loyalty, the PC had better work toward it. C'mon, at least show up at the smith's funeral and offer your condolences to the family. Offer a break in the taxes until the family get's business rolling again. Do the things that make your followers like you!
Followers do not represent free labor. Followers do not represent undying fanatacism. Followers represent common people that admire and have loyalty to somebody that might otherwise appear to be another sword/wand/fist waving mercenary. They give a PC credibility. They look out for the PC's interests within their capability.
Old Farmer Joe might come up to the manor house one day. He was out tillin' his fields and this here rough lookin' young'un with a sword stopped by and was askin' questions. Seemed kinda mad at you fer sumthin. Anyway, he took off down the road and disappeared into the trees a little ways off. I know ye gots your own business, but you done right by me when you kept those goblins from messin' up my barn. And you had your cleric friend stop by that one winter when I broke my leg. Your a good guy and I like havin' you as the lord here in these parts. So, maybe it ain't nuthin' to worry about, but if that feller came lookin' fer you and I didn't tell you, I just wouldn't feel right about it.
Old Farmer Joe didn't demand that the ruffian leave. He didn't ask too many questions. He wouldn't rush out and attack the guy. Why? Because he is, essentially, a commoner. It isn't in his capability to do that. If it was, he wouldn't be farming. But, he has loyalty. Something that no good leader will undervalue. While others might have shrugged off the encounter and gone about their business, Old Farmer Joe felt like he needed to warn the PC of a possible threat. The PC has a loyal follower, that is adding to his coffers a little each year. You have a story element and a plot delivery device.
Of course, when the PC decides that he wants to capture a new castle and move his town, Old Farmer Joe might not want to go. Why? Because he already has a home, and a family. Why would he want to move? Maybe some followers would move, but the PC will have to setup his new keep and start making friends with the locals to "refill" his allotment of followers over the next few seasons.
I'm sure this is not how Players would like to see followers being used. They would probably prefer a loyal encampment that is willing to do anything for the PC at the drop of a hat. But, that is not what the feat says you get. And in my game, that's not what you end up with.