Bacon Bits
Legend
From the DMG, p.84 "Assuming typical adventuring conditions* and average luck, most adventuring parties can handle about six to eight medium or hard encounters in a day," and a little later, "This provides a rough estimate of the adjusted XP value for encounters the party can handle before the characters will need to take a long rest."
The reason that 6-8 encounters per adventuring day figure gets tossed around is because people interpret "can" as "should."
Either way, though, this is the tail wagging the dog.
They didn't write Chapter 3 of the DMG first. They decidedly wrote it last. That's why it came out in December, while the PHB had an August release and the MM was released in September. If you remember, WotC kind of made it clear that they didn't really know how to plan encounters when the PHB and MM were released because those rules hadn't been finalized yet. That's why there's no basic framework at all in the PHB or MM for encounter building, and nothing on WotC's website for doing the same thing. That means the DMG and the encounter building rules were written after the Experience Points by Challenge Rating table on MM p9, and after the Character Advancement table on PHB p15.
However, those two tables by themselves don't give you enough information to create the XP Thresholds By Character Level on DMG p82, or the Adventuring Day XP table on DMG p84.
What Crawford had to do is sort of work backwards from the above two tables and knowing how quickly they wanted PCs to advance. Then he had to decide how difficult an encounter should be. Like you look at the XP Thresholds By Character Level table, and there's nothing obvious that says X is what he used for Easy, Y is Medium, etc. If the design idea holds water -- that Challenge represents how well effectively resources are consumed -- then the relative columns on that table are completely arbitrary. There's no reason that "Easy" couldn't start at 50 XP at level 1. It's really an arbitrary breakdown where you're putting a pin into a board somewhere and saying, "This is how many encounters we want per day."
I think what Crawford did was set the encounter difficulty very low. I think the primary motivation for doing this is two-fold:
His first reason was to encourage groups that don't prefer combat to play the game. Some groups don't care about combat at all, basically. They want a more narrative-driven game. It's easier to do that if the story pacing isn't thrown off by the constant need to stop and do nothing for 15 hours, then sleep for 8 hours, and only then be able to continue on your way. One of the big complaints about 4e (yes, and 3.x, but the point is that it survived into 4e) was the "5 minute adventuring day," and I think in a lot of ways this came about because of the encounter difficulty and general tankyness of everything in 4e. When I played 4e, I felt like I had to use at least 1 Daily power every encounter. Maybe WotC decided that meant encounters were too difficult in general, or maybe they got a complaint that 4e was too complex in general and they decided that contributed to the poor player retention rates for 4e.
This reason, by and large, is pretty irrelevant because it doesn't have any mechanical game impact. An individual group will always find whatever level of difficulty that they want for their encounters. That's what the range exists on the XP Thresholds by Character Level for.
Crawford's second reason is because short rests have to happen. What really seems hidden by the game is that -- as a DM -- you're really supposed to be designing your encounters to allow short resting as often as possible. At least once a day. If short rests don't happen, then Fighter and Warlock get pretty shafted because they run out of abilities, Monk can get kind of shafted without Ki points early on, etc. So now the game needs to be built in a way that your PCs want to take short rests, and have the ability to take short rests. That means encounters need to be built in such a way that enough resources get consumed that you need to short rest, but not enough resources get consumed that you must long rest. The easiest way to guarantee that groups always hit this short rest threshold and don't meet the long rest threshold is to use weaker and more numerous encounters. That makes resource expenditure more granular, which means the PCs should hit short rests more often. So, you instruct DMs to nickel and dime the party with weaker encounters. In other words, DMG "Deadly" is actually closer to Medium-Hard, DMG "Hard" is actually closer to Medium, DMG "Medium" is really Easy, and so on.
This... presents a real problem. Some groups like hard combat, or just want fewer combats. If groups want to decrease the number of encounters and increase their difficulty to meet their Adventuring Day budget, well, they have to do that at the expense of certain class abilities. This is somewhat of a new phenomenon in D&D. Previously, you could adjust encounter difficulty as you wanted. The only problems you'd introduce are exactly what you'd expect: TPKs and faster advancement. Now, however, harder encounters risk pushing one or more PCs to the long rest threshold -- the point where they don't want to short rest because they can't meaningfully continue adventuring. So, what does the party do when short rest PCs often want to short rest, long rest PCs don't need to short rest, and encounters are hard and by their very nature resources aren't spent evenly across the board.
Now, yes, the DM can add contrived reasons that the party can short rest and not long rest, but now we're restricting adventure design to get the mechanics of the game to behave the way we want. And, yes, we do that already, but that doesn't mean doing it more is good for the game. Furthermore, we're restricting adventure design further than what previous editions did, so in theory there may be adventures that are much more difficult to adapt to 5e because there isn't the kind of encounter rate that you'd need to have for everybody to feel satisfied.
This strikes me as a pretty fundamental and unavoidable design issue in 5e. I hesitate to label it a flaw, since I think some groups will find the outcome favorable. However, the groups I've played in definitely have not found it favorable. The real issue is that it causes arguments within the party. I imagine that the design was to cause that tension to make "interesting choices," but I really don't find this choice particularly interesting. Fighter wants to rest and recover all his spent abilities because he's stuck on auto attack. Rogue wants to keep going because he avoided damage and has no short rest abilities. Cleric kind of wants to short rest because he's spent his short rest recovery, but doesn't mind continuing because long rest is fine. Barbarian wants to long rest because she took a beating in the last encounter, and he knows that if she short rests she won't be able to recover all of her Hit Dice with the next long rest, meaning she'd be short healing for tomorrow. This complexity leads to party indecision and party arguments. Except for towns, the only time I've seen my group decide to do different things (i.e., split up) has been when they wanted different rest schedules. If your game cooperative game design causes the players to argue... that's a problem!
Overall I find this incredibly frustrating because I think most people would agree that, in previous editions, 2-4 encounters per day was standard. I think most groups want to continue to play that way, and I think most groups are fairly unwilling to move away from that and doing so feels rather awkward. So not only is 6-8 an arbitrary number to encourage short rests, it feels excessive to people already familiar with the game. 4e's healing surge and rest system much better handled this issue, as did 3e, and I'd go so far as to say that even 1e/2e handled it better. I don't ever remember arguing about what to do in those editions.
I think this is one of the reasons that some of the most common house rules I've seen modify the rest schedule directly. Some say, "short rests are once a day, long rests are once a week." Others say, "short rests just require time, long rests require comfort of an inn." Those work around the issue by forcing short rests to be actually necessary to gameplay. However, I'm not convinced they actually help, or they just "solve" the problem because the decision is taken away from the PCs. However, the underlying resource scarcity problem is still there. Maybe that increased scarcity is desirable, but it's decidedly a major change.
I expect the knee-jerk response here is going to be some form of, "Just play the game like we do and you won't have this trouble." Well... no. That's not an acceptable solution. My groups have played D&D the way we play 5e for 20 years. Every edition of the game -- including 4e -- has supported our play style. If 5e suddenly doesn't support our playstyle, that's not my group's fault. That tells me that the designers changed something that they should not have. D&D is a game that's intended to be played the way the table wants to play it. If the table runs into problems just by changing the encounter rate and encounter difficulty, I consider that a very significant problems. There's no magic solution to the short/long rest dichotomy except to fundamentally alter the rest schedule or eliminate one or the other type of ability.
"Ok, well, just don't play 5e." Well, what happens when 6e comes out? Won't it be the same issue if everybody who has this problem abandons the game? We like 5e. We like 5e's content, too. 5e is our game as much as it is anybody's, and dismissing us because we have a problem that you don't is patronizing. Except for this issue, 5e is clearly the best version of D&D. It just has this hole in the middle. The hole is not as big as 1e multiclassing and obtuse mechanics, 2e THAC0 and multiclassing, 3e QWLF and item creation, or 4e radical mechanics and mathematic treadmill, but -- with the exception of 4e -- none of these previous issues have caused argument or division at our table.