I think the gold standard for base building is Mutant Year Zero, though it would take some effort to adapt it to D&D.
In MYZ, the PCs are residents of "The Ark", a settlement of maybe 150 mutants. This settlement has its capabilities measured with four different values: defense, technology, culture, and food. At the start of the campaign, all four stats are pretty low.
Every session, the PCs decide on what, if any, improvement to work on. These are usually, but not always, physical in-game things but can also be social advances. In-world, it's probably a decision made by the Ark's various bosses, and the GM can override the PCs because "No, you just got hit by raiders, the people want something that increases defense", but it's basically left to the players. Building an improvement requires a certain number of successful skill checks on behalf of the PCs – much like the decision-making, while the actual construction is a collective effort it's based on the success of the PCs. Most improvements increase one or more of the settlement's stats which in turn provides distinct benefits – building a pig farm increases the settlement's food stat, and when it becomes high enough food and water becomes cheaper. Building a statue increases culture, which makes the Ark more harmonious.
Something similar could be done in D&D. Building/improving a smithy might unlock the ability to buy masterwork weapons. A temple provides a place for healing. An inn gives you a never-ending source of mysterious robed figures selling treasure maps of questionable provenance. And so on. This kind of thing sounds more interesting to me than building a castle and raising an army (even though the settlement's defenses are definitely things that could be improved).