I went for Moderate, but my players know me well enough by now that they will expect to run into things they really should just hide from. I have no interest in catching them out, though, so such things are usually well-telegraphed.
Also, the difficulty of any wilderness is proportional to the role it plays in the adventure. In some adventures, the wilderness *is* the adventure, whereas in others, it can easily get in the way of the action. I won't be rolling for random encounters if a month's worth of meticulous pacing is going to pay off a day's travel away.
That sound you can hear is the sharp intake of breath from a thousand sandbox DM's.