[MENTION=73976]Calion[/MENTION], Let me see if I get this right, you have no experience living or surviving in the wild and you take Survivorman (a psuedo-reality entertainment product) as your souce of truth?
One person with wilderness experience has said he has lived in the wilds at "Comfortable" living standard. I will add that have have spent upto 8 consecutive weeks living off my skills in the wilds. I was comfortable, never more hungry, hot or cold then when I live my comfortable life at home with my A/C and car. For someone proficient, it's not big deal.
More importantly, as has been stated, the down-time rules are there to make down-time quick and simple. It is not intended to be a reality simulator, nor are the intended to be detailed. It's not what the game is about.
In the end, if you don't think the rules fit your "reality" then change them when you DM.
How many changes of clothes did you have? Did you eat three good meals a day, every day, with large variety in the dishes you ate? Did you have a soft bed and other furnishings? Reasonably reliable light (equivalent to candles or oil lamps)? What legal protections did you have (oh wait, you had plenty of those. That's probably not the case in a D&D environment). Were you safe from wild animals (yes, probably, in this day and age. Not true of adventurers) and monsters?
You certainly can have been "comfortable" relative to someone without your skills. I don't see how you can have been as comfortable as someone living in a private room at a fine inn in 1200 A.D.—unless you take into account your enjoyment of the outdoors, which does not count here.