If it's downtime, what's the difference?
Well, I consider downtime to just be days that aren't specifically the "adventure." I've played in games where 99% of the games were downtime, then 1% was the adventure.
And yes, I consider travel days to be downtime. And if an obstacle comes around, it's just an encounter during downtime up until the adventure location has been arrived. If there's a string of encounters during travel, then it's part of the adventure.
So when your party spends the day at the minister's mansion to get to know all the possible suspects of the crime and no real combat occurs, that's downtime.
When your party spends a day searching the empty ruin for information, that's downtime.
When your party spends the day gathering rumors, that's downtime.
Even the very first tavern day, when all the characters introduce themselves, is a downtime day if all they do is get the quest information and head off the next day.
Surely, these days are important to the overall adventure, yet still has the time-tolerance for a warlock to cast their spells and short rest.