It depends on my character. My previous character spent downtime romancing an NPC, or researching how to get home or rumors possibly related to events we happened to participate in, or trying to navigate the government and aristocracy in an attempt to find a diplomatic solution to the assassin plagueing his life.
My current character is based on Redgar and a friend in real life, (though he doesn't know it) and thus follows the idea, "why should I keep track of treasure, I'm just going to spend it all on Ale and Whores" and I have stuck to that principle as much as is safely possible. Despite having the Ancestral Weapon feat and 10,000 gold pieces worth of material to sacrifice to power my ancestral weapon up, I've had my character spend his free time at a brothel. I did buy some healing potions before setting out on the adventure again. I tried to gather some information this last visit, since the campaign is city based, (same city as the previous campaign and character), but with a Charisma of 8 it's no wonder he has to pay for attention.
With my other gaming group my ogre frequently spent his downtime doing absolutely nutty stuff like digging a moat, building a woodland fortress and organizing a pig wrestling contest.