Well, castles cost a lot. And you have to pay followers, or otherwise see to their well-being.
But my question was why can't you have things that the characters care about. The OP seemed to be noting, basically, that after a while, fighting for the sake of fighting gets boring, so give the PCs something else to deal with.
I am questioning why "build a castle and get followers" is really the only "thing they care about" on the table here. It is, honestly, kind of specific, and probably doesn't apply to a lot of PCs.