That last is what tends to ruin these things: that you never get to go there.
That said, was it because the next adventure always came on too fast, or because your group didn't tend to split up during downtime?
If it's because the group doesn't split up to pursue independent downtime activities, maybe suggest a common stronghold for the party.
So, my early AD&D campaigns didn't have stable "adventuring parties"- everyone had their own characters, and they decided who they were playing, and they all sort of just ended up on the adventure du jour, lol. People could get solo time and even solo adventures, if the DM had time for them, but because I had to go where the adventure was, by the time I had a home base to call my own, the adventure was rarely anywhere near it.
As time went on, there was a shift in the way people ran games. Suddenly you had stable adventuring parties that went on adventures with one another, and it was always one adventure to the next. Downtime was a rarity, because there was always something to do.
When we did have downtime, it was usually the dreaded "city session" where the party suddenly splinters and goes shopping, and you basically had to sit around and twiddle your thumbs until the DM got back to you, lol.
Most DM's never really knew what to do with downtime, and when the WotC era suddenly introduced cool things spellcasters could do with downtime (like creating magic items), a new problem erupted.
Wizzo the Wondrous decides "hey, we got some money, I can make us a few magic items."
Party: "Huzzah!"
Wizzo rents his space and details his plans. The rest of the party, having nothing to do with this process, start asking the DM what they can do during this time. This resulted in either a lot of nothing, since the DM had no real plans (inevitably, the warriors would find some pit fight or whatever to try and earn a few coins or just end up getting arrested lol), or worse, they manage to convince the DM in giving them a solo adventure, which makes Wizzo's player very annoyed that he's basically giving up play time to crank out some magic swag for the party members!
The last Pathfinder 1e game I was involved with, the party up and decided to take a mission to sail to another town, ditching the spellcaster while he was enchanting things for them! Upon learning this, that player left the game, and the magic items (and the party's loot) were never seen or heard from again!
I was absent for this session, so I lost treasure over this through no fault of my own so I had pretty mixed feelings about it. On the one hand, the other players deserved their fate, but on the other hand, I don't think the GM should have allowed the scenario to play out- from an in-universe stance, characters can do what they will, sure, but the reality is, one person was excluded from the game because they dared to invest personal resources (feats, skill points, etc.) to support the party.
Anyways, for my current campaign, I'm putting a lot more effort into giving every player something interesting to do during downtime, and presenting interesting downtime activities, though even with that, I've noticed that unless the benefit is really good, they still would rather adventure than sit around for a few weeks, lol.
In the previous 5e game, the only character who really seemed to want or need downtime was the Wizard, and even that was simply to scribe spells. The other players mostly screwed around until it was time to get back to the adventure.
That's why I'm thinking maybe a fun base building minigame will make the players more invested in this sort of thing.