Those other 100 followers are effectively henchmen and hirelings - guardsmen, apprentices, servants, and whatnot. They're not really intended to come along on adventures.
I'm not planning on the followers coming along on adventures either, and will all be 4e minions. But I still think they can do other things than just exist, which is why I want to use them as part of the treasure of a stronghold space. So if a rogue has a hideout for example, he can have a follower who is a spy and allow him to reroll a streetwise check.
Basically, I'd like each minor follower (or unit of followers) to do something that has a tangible game benefit, and act like a magical item with daily powers. That's why I'd have followers and stronghold components work like treasure and magical items. It would be nice to pick up followers in the course of your adventures among the monsters and NPC's. A successful diplomacy challenge could convince him/her/it to come work for you.
As far as developing/building the stronghold... I would say the player would have to choose a site and build something from scratch, or find something suitable and "fix it up" himself. The first option would cost a bit more, but enable greater customization (i.e. better bonuses and possibly easier advancement); the second, obviously, would be cheaper but more difficult to customize.
I was kind of hoping that would be easier to do as treasure. For example, the DM could point out that the alchemical equipment found in a dungeon could be disassembled and taken back to their stronghold to improve their own magical laboratory. A stronghold taken from an orcish tribe could be a lost ducal stronghold from the fallen kingdom of Nerath, a match for any stronghold available today.
Sure it will probably lead to players hauling art objects and furniture and everything else not nailed down back to their strongholds, but I think that's part of the fun of D&D.
You can make it modular, so that groups who aren't really interested in running a domain can go off adventuring and let the followers "mind the store", while others interested in politics and intrigue can play at being a lord.
That's exactly what I am thinking too.