D&D 5E D&D Next playtest post mortem by Mike Mearls and Rodney Thompson. From seven years ago.
Mistwell
Crusty Old Meatwad (he/him)
Video #3 [Open Playtest]
They read all the comments.
The surveys were extremely important to the final product. Absolutely essential.
They figured out out to best iterate with a purpose on problems later in the process and wish they had figured that out earlier in the process.
People's ratings of the game went up and up with each packet of the playtest. There were minimum quantity thresholds for feedback before data could be considered sufficient to consider.
The forums are not necessarily representative of the larger audience. There definitely is a silent majority sometimes. A lot of times people would say something is terrible on forums, that came back with 95% approval on surveys. It was most useful to use forums when the forum views lined up with the survey data, where they could then ask forum people more about that aspect of the game. Also sometimes the thing people would complain about was more a sign something was going wrong in a broader issue, and not what they would specifically complain about. Like for example someone might complain about not hitting enough bad guys, but they'd find that was really a symptom of a cause of lack of sufficient movement for PCs.
Here is an image of a sample of some of their data, measured on a 1-5 scale of satisfation. NOTE: They specify the headers are off on this.all three columns are off (if you can even read them in this grainy image - supposed to be combat satisfaction, non combat satisfaction, and average satisfaction overall, but numbers may not be in that order). This was not the headers they worked with he just knew they got moved before the presentation by accident:
Here is another on Class Complexity from a prior set of data:
There was another chart here on Complexity vs Non-Combat satisfaction and they found classes high on non-combat satisfaction tended to also be high on complexity. So for non-combat, people liked the complex classes.
The reverse was true for combat. Audiences said they were having the least amount of fun with the more complex classes in combat, and the most amount of fun with the least complex classes in combat.
This set of conclusions was the complete reverse of the designer initial assumptions about the game. The assumption was that during combat people liked having a lot of options, and that during non-combat they wanted a lot more free form and not as many mechanical options. It turned out it seemed to be a time pressure issue. When out of combat, planning stuff was more fun with lots of options to work with and time to flip through books and such. But once in combat and everyone is waiting for you to complete your turn you don't want to have too many options and flipping through books resulting in slowing the game down.
Here is an image of Rogue Satisfaction from October 2012. I am sorry the image is so grainy I couldn't seem to grab a clearer one.
Cunning Action was being rated at 100% (40.1% apparently rated it top, and it had the top rating of being above everything else) as the top rated element of the rogue. Cunning Action was at the time of this survey part of the Thief subclass and not part of the core Rogue abilities. Thieves Cant was getting the lowest rating at 3.1% (which they didn't care much about because they viewed it as a ribbon mechanic.)
They followed this up with a grainy image of two different Rogue surveys (August and October) charted against each other showing where there were response improvements from changed in the playtest for that class, and still some dips that fell below their threshold for acceptable that continued to need work.
They read all the comments.
The surveys were extremely important to the final product. Absolutely essential.
They figured out out to best iterate with a purpose on problems later in the process and wish they had figured that out earlier in the process.
People's ratings of the game went up and up with each packet of the playtest. There were minimum quantity thresholds for feedback before data could be considered sufficient to consider.
The forums are not necessarily representative of the larger audience. There definitely is a silent majority sometimes. A lot of times people would say something is terrible on forums, that came back with 95% approval on surveys. It was most useful to use forums when the forum views lined up with the survey data, where they could then ask forum people more about that aspect of the game. Also sometimes the thing people would complain about was more a sign something was going wrong in a broader issue, and not what they would specifically complain about. Like for example someone might complain about not hitting enough bad guys, but they'd find that was really a symptom of a cause of lack of sufficient movement for PCs.
Here is an image of a sample of some of their data, measured on a 1-5 scale of satisfation. NOTE: They specify the headers are off on this.all three columns are off (if you can even read them in this grainy image - supposed to be combat satisfaction, non combat satisfaction, and average satisfaction overall, but numbers may not be in that order). This was not the headers they worked with he just knew they got moved before the presentation by accident:
Here is another on Class Complexity from a prior set of data:
There was another chart here on Complexity vs Non-Combat satisfaction and they found classes high on non-combat satisfaction tended to also be high on complexity. So for non-combat, people liked the complex classes.
The reverse was true for combat. Audiences said they were having the least amount of fun with the more complex classes in combat, and the most amount of fun with the least complex classes in combat.
This set of conclusions was the complete reverse of the designer initial assumptions about the game. The assumption was that during combat people liked having a lot of options, and that during non-combat they wanted a lot more free form and not as many mechanical options. It turned out it seemed to be a time pressure issue. When out of combat, planning stuff was more fun with lots of options to work with and time to flip through books and such. But once in combat and everyone is waiting for you to complete your turn you don't want to have too many options and flipping through books resulting in slowing the game down.
Here is an image of Rogue Satisfaction from October 2012. I am sorry the image is so grainy I couldn't seem to grab a clearer one.
Cunning Action was being rated at 100% (40.1% apparently rated it top, and it had the top rating of being above everything else) as the top rated element of the rogue. Cunning Action was at the time of this survey part of the Thief subclass and not part of the core Rogue abilities. Thieves Cant was getting the lowest rating at 3.1% (which they didn't care much about because they viewed it as a ribbon mechanic.)
They followed this up with a grainy image of two different Rogue surveys (August and October) charted against each other showing where there were response improvements from changed in the playtest for that class, and still some dips that fell below their threshold for acceptable that continued to need work.
Oofta
Legend
The fact that the playtest for the anniversary edition had multiple kinds of elf was something we were joking about yesterday during our game session. I don't think having an elf of the week is a good thing. Whether or not more classes or options would have been beneficial is debatable.
Mistwell
Crusty Old Meatwad (he/him)
Video #4
Starts with Druid satisfaction data (very grainy). People were not very satisfied with the druid during these earlier surveys. Wildshape had negative reactions. Druid wasn't changing to the positive between two surveys, unlike the Rogue.
Here was their focus on Wildshape at that point:
Some perceptions of satisfaction changes with experience playing the game as well. Druid went down in satisfaction after the first year. Similar to how there was a common belief Monk was overpowered on 3e first coming out, and then seen as underpowered over time. Similar with 3e Spiked Chain was viewed as weak early, and strong later.
The smaller group of Alpha playtesters under NDAs had tracking specific to them, particularly with PHB material. This chart showed what portion of their feedback resulted in actionable changes, or did not.
Next slide had just these three points: This that didn't go well with the playtests: 1) Too many changes at once, 2) All testers must re-learn the game, and 3) Difficult to pinpoint problems.
Next slide another three points: Things that worked: Diversity 1) Demolish Assumptions, 2) In-World Art Descriptions, and 3) Personality, Not Stereotype. Improve how women portrayed. Show different ethnicities and cultures. Specify more detail, with an art bible.
Starts with Druid satisfaction data (very grainy). People were not very satisfied with the druid during these earlier surveys. Wildshape had negative reactions. Druid wasn't changing to the positive between two surveys, unlike the Rogue.
Here was their focus on Wildshape at that point:
Some perceptions of satisfaction changes with experience playing the game as well. Druid went down in satisfaction after the first year. Similar to how there was a common belief Monk was overpowered on 3e first coming out, and then seen as underpowered over time. Similar with 3e Spiked Chain was viewed as weak early, and strong later.
The smaller group of Alpha playtesters under NDAs had tracking specific to them, particularly with PHB material. This chart showed what portion of their feedback resulted in actionable changes, or did not.
Next slide had just these three points: This that didn't go well with the playtests: 1) Too many changes at once, 2) All testers must re-learn the game, and 3) Difficult to pinpoint problems.
Next slide another three points: Things that worked: Diversity 1) Demolish Assumptions, 2) In-World Art Descriptions, and 3) Personality, Not Stereotype. Improve how women portrayed. Show different ethnicities and cultures. Specify more detail, with an art bible.
Mistwell
Crusty Old Meatwad (he/him)
This is addressed specifically in the next slide - turned out when people were presented with fewer out of combat options and more in combat options with actual rules, they liked the opposite due to time pressure constraints on combat and no pressure constrains on out of combat.
The point was some debates in the playtest were made while others weren't.
For example, the Orc is being approached as a Player option in the PHB in the 2022 playtest. That was something I believe not even attempted in the 2013 open playtest. Likely because the idea strayed away for past versions of D&D a lot.
Parmandur
Book-Friend
I knew they said this at some point, just couldn't recall where. Explains a lot of what goes down in conversations here.
Mistwell
Crusty Old Meatwad (he/him)
Video #5 Questions
How did they choose which playstyles to support: they wanted enough optimization to satisfy those who wanted high level customization, but not too far in that direction because that moved away from the center (majority view) of the game. They wanted it to be at least everyone's second best version of the game. Making feats an optional rule for example was a compromise they made. The DMG was intended as a tool chest for optional rules.
Question about playtesting and why it's not as sophisticated in RPGs as it is with video games. Mearls says it's about the deadlines and a continual release schedule. They found in one of their playtests how many books people would buy for expansion books. They thought it would be 20 or 30 in ten years, and it turned out to be FOUR. FOUR was the average number of expansion books people bought over ten years. That changed their release schedule. They made them to decide to scale back the number of releases and focus more on playtesting. They looked back and realized they had saturated the market with the release schedule in 3.5 and 4e. They wanted to avoid the treadmill, and that the old business model of RPG markets was outdated. Mearls compared this to the board game industry.
The playtesting method had been tried by WOTC on the Dungeon Command board game prior to 5e, as a test. They learned some process lessons from that.
[This is at 6:26 mark in the video if you want to watch the details - pretty interesting stuff].
[I had to end this viewing at the half way mark at 12:45 due to time. Will continue later.]
How did they choose which playstyles to support: they wanted enough optimization to satisfy those who wanted high level customization, but not too far in that direction because that moved away from the center (majority view) of the game. They wanted it to be at least everyone's second best version of the game. Making feats an optional rule for example was a compromise they made. The DMG was intended as a tool chest for optional rules.
Question about playtesting and why it's not as sophisticated in RPGs as it is with video games. Mearls says it's about the deadlines and a continual release schedule. They found in one of their playtests how many books people would buy for expansion books. They thought it would be 20 or 30 in ten years, and it turned out to be FOUR. FOUR was the average number of expansion books people bought over ten years. That changed their release schedule. They made them to decide to scale back the number of releases and focus more on playtesting. They looked back and realized they had saturated the market with the release schedule in 3.5 and 4e. They wanted to avoid the treadmill, and that the old business model of RPG markets was outdated. Mearls compared this to the board game industry.
The playtesting method had been tried by WOTC on the Dungeon Command board game prior to 5e, as a test. They learned some process lessons from that.
[This is at 6:26 mark in the video if you want to watch the details - pretty interesting stuff].
[I had to end this viewing at the half way mark at 12:45 due to time. Will continue later.]
Parmandur
Book-Friend
Wow, so actually for most people the 5E product release schedule has bern blisteringy fast.
GMforPowergamers
Legend
complex martial
Similar Threads
D&D (2024)
D&D playtest feed back report, UA8
