Certainly, if a character wants to take downtime, and the others want to adventure then of course the character who wants downtime doesn't have to go on the adventure. They just go do whatever they want to do, eg exercise class abilities and laze about, instead of adventure.
This is pretty much...