darjr
I crit!
I'm surprised this hasn't been posted before. At PAX Mike and Rodney gave a post mortem about the D&D Next playtest. It's in seven parts about twenty minutes a piece. It's pretty detailed and discusses what they found and what they did based upon their interoperation of the data.
I think it's important to baseline how they took the Next playtest results, and in my opinion, it gives us a view into how they've approached playtesting ever since. This to me is a very interesting deep view on what the playtest resulted in internal to WotC.
Interesting bits for me are they played all versions of D&D before starting and took detailed notes and they discuss how they worked together. They noted that they read every comment of the surveys and they DO read the forums, but not how you might think, also there was an Alpha test NDA group outside of the public playtesters too.
There is so very much here I'm shocked that the view numbers for these videos are so low and that I haven't heard of them till now, that they haven't been picked apparat and studied in detail. I think these would have been especially valuable to early third party adopters.
the insight about their decisions based upon the playtest is interesting as well for instance moving the bonus abilities from the thief class to the base rogue class. Also the bit about them thinking that complex characters would be more fun at the table being completely opposite from what the playtesters found.
They also talk about the art and diversity in the game, from early on they were at least thinking about it and working towards it.
Ultimately the thing about the playtest is that it can satisfy two opposing things in D&D game design, the need to get as much good input as possible with as small of a design team as possible and no smaller.
I haven't listened to all of it yet and I want to give it another listen. I'll note that the auto transcript of youtube is also helpful.
Note they uploaded this seven years ago.
This is a playlist with all seven videos.
Mistwell put together a summary of the videos with some slides captured and inserted! Thanks!
Go like his post.
I think it's important to baseline how they took the Next playtest results, and in my opinion, it gives us a view into how they've approached playtesting ever since. This to me is a very interesting deep view on what the playtest resulted in internal to WotC.
Interesting bits for me are they played all versions of D&D before starting and took detailed notes and they discuss how they worked together. They noted that they read every comment of the surveys and they DO read the forums, but not how you might think, also there was an Alpha test NDA group outside of the public playtesters too.
There is so very much here I'm shocked that the view numbers for these videos are so low and that I haven't heard of them till now, that they haven't been picked apparat and studied in detail. I think these would have been especially valuable to early third party adopters.
the insight about their decisions based upon the playtest is interesting as well for instance moving the bonus abilities from the thief class to the base rogue class. Also the bit about them thinking that complex characters would be more fun at the table being completely opposite from what the playtesters found.
They also talk about the art and diversity in the game, from early on they were at least thinking about it and working towards it.
Ultimately the thing about the playtest is that it can satisfy two opposing things in D&D game design, the need to get as much good input as possible with as small of a design team as possible and no smaller.
I haven't listened to all of it yet and I want to give it another listen. I'll note that the auto transcript of youtube is also helpful.
Note they uploaded this seven years ago.
This is a playlist with all seven videos.
Mistwell put together a summary of the videos with some slides captured and inserted! Thanks!
Go like his post.
Here is my complete review of these videos compiled in one post for convenience.
Here are all the links to the Mike Mearls and Rodney Thompson videos
Link 1
Link 2
Link 3
Link 4
Link 5
Link 6
Link 7
Here is my summary of each video (which varies in the detail I go into over time):
Video #1
They learned from feedback that they didn't need an equal number of rules for each of the three pillars of the game.
For example, for Social Interaction, the audience feedback was that the audience didn't want a lot of game mechanics governing that pillar. They wanted to role play out their character, maybe talk in funny voices, maybe make a charisma check, but they didn't need a whole of [of mechanics] to feel satisfied with it. Character personality on your character sheet intentionally takes up about as much space as combat on your character sheet, and those personality traits are supposed to help serve as a reminder and support for social interactions, and to note it's an important concept to players.
For exploration, they presented material in a different way. They don't give players specific options like "You can make a perception check" but instead focus a lot more on you just telling the DM what you want to do and the DM adjudicates that.
Video #2
Having a smaller design team was intentional, and a strength. [Some of this is my interpretation of what they're saying here] Too many people requires a lot more communication to get everyone on the same page for evolving design standards, and a lot of time was spent in prior design teams either communicating more or fixing stuff that hadn't been communicated well. Fewer internal people made for the vision to be a lot more consistent across the rules.
Making the books fun to read was an important idea.
D&D isn't as hard as people think it is to play.
Video #3 [Open Playtest]
They read all the comments.
The surveys were extremely important to the final product. Absolutely essential.
They figured out out to best iterate with a purpose on problems later in the process and wish they had figured that out earlier in the process.
People's ratings of the game went up and up with each packet of the playtest. There were minimum quantity thresholds for feedback before data could be considered sufficient to consider.
The forums are not necessarily representative of the larger audience. There definitely is a silent majority sometimes. A lot of times people would say something is terrible on forums, that came back with 95% approval on surveys. It was most useful to use forums when the forum views lined up with the survey data, where they could then ask forum people more about that aspect of the game. Also sometimes the thing people would complain about was more a sign something was going wrong in a broader issue, and not what they would specifically complain about. Like for example someone might complain about not hitting enough bad guys, but they'd find that was really a symptom of a cause of lack of sufficient movement for PCs.
Here is an image of a sample of some of their data, measured on a 1-5 scale of satisfation. NOTE: They specify the headers are off on this.all three columns are off (if you can even read them in this grainy image - supposed to be combat satisfaction, non combat satisfaction, and average satisfaction overall, but numbers may not be in that order). This was not the headers they worked with he just knew they got moved before the presentation by accident:
Here is another on Class Complexity from a prior set of data:
There was another chart here on Complexity vs Non-Combat satisfaction and they found classes high on non-combat satisfaction tended to also be high on complexity. So for non-combat, people liked the complex classes.
The reverse was true for combat. Audiences said they were having the least amount of fun with the more complex classes in combat, and the most amount of fun with the least complex classes in combat.
This set of conclusions was the complete reverse of the designer initial assumptions about the game. The assumption was that during combat people liked having a lot of options, and that during non-combat they wanted a lot more free form and not as many mechanical options. It turned out it seemed to be a time pressure issue. When out of combat, planning stuff was more fun with lots of options to work with and time to flip through books and such. But once in combat and everyone is waiting for you to complete your turn you don't want to have too many options and flipping through books resulting in slowing the game down.
Here is an image of Rogue Satisfaction from October 2012. I am sorry the image is so grainy I couldn't seem to grab a clearer one.
Cunning Action was being rated at 100% (40.1% apparently rated it top, and it had the top rating of being above everything else) as the top rated element of the rogue. Cunning Action was at the time of this survey part of the Thief subclass and not part of the core Rogue abilities. Thieves Cant was getting the lowest rating at 3.1% (which they didn't care much about because they viewed it as a ribbon mechanic.)
They followed this up with a grainy image of two different Rogue surveys (August and October) charted against each other showing where there were response improvements from changed in the playtest for that class, and still some dips that fell below their threshold for acceptable that continued to need work.
Video #4
Starts with Druid satisfaction data (very grainy). People were not very satisfied with the druid during these earlier surveys. Wildshape had negative reactions. Druid wasn't changing to the positive between two surveys, unlike the Rogue.
Here was their focus on Wildshape at that point:
Some perceptions of satisfaction changes with experience playing the game as well. Druid went down in satisfaction after the first year. Similar to how there was a common belief Monk was overpowered on 3e first coming out, and then seen as underpowered over time. Similar with 3e Spiked Chain was viewed as weak early, and strong later.
The smaller group of Alpha playtesters under NDAs had tracking specific to them, particularly with PHB material. This chart showed what portion of their feedback resulted in actionable changes, or did not.
Next slide had just these three points: This that didn't go well with the playtests: 1) Too many changes at once, 2) All testers must re-learn the game, and 3) Difficult to pinpoint problems.
Next slide another three points: Things that worked: Diversity 1) Demolish Assumptions, 2) In-World Art Descriptions, and 3) Personality, Not Stereotype. Improve how women portrayed. Show different ethnicities and cultures. Specify more detail, with an art bible.
Video #5 Questions
How did they choose which playstyles to support: they wanted enough optimization to satisfy those who wanted high level customization, but not too far in that direction because that moved away from the center (majority view) of the game. They wanted it to be at least everyone's second best version of the game. Making feats an optional rule for example was a compromise they made. The DMG was intended as a tool chest for optional rules.
Question about playtesting and why it's not as sophisticated in RPGs as it is with video games. Mearls says it's about the deadlines and a continual release schedule. They found in one of their playtests how many books people would buy for expansion books. They thought it would be 20 or 30 in ten years, and it turned out to be FOUR. FOUR was the average number of expansion books people bought over ten years. That changed their release schedule. They made them to decide to scale back the number of releases and focus more on playtesting. They looked back and realized they had saturated the market with the release schedule in 3.5 and 4e. They wanted to avoid the treadmill, and that the old business model of RPG markets was outdated. Mearls compared this to the board game industry.
The playtesting method had been tried by WOTC on the Dungeon Command board game prior to 5e, as a test. They learned some process lessons from that.
[This is at 6:26 mark in the video if you want to watch the details - pretty interesting stuff].
Question regarding errata issues for balance topics: First they would release a playtest document, likely multiple times with interactions. Then they would likely issue a new subclass which addresses the issues which givens DMs the option to replace the older subclass. Then they could offer a new class option for the class, as an optional rule. [Note: These are things they did in Tasha's, but were considering it as a means of addressing issues all the way back in 2015!]
On conversion guides for converting an older campaign from a prior edition to 5e: They are working on conversion guides. The DMG has levelled treasure packages as a guideline.
If a person wants to continue with prior edition character that has a lot more complexity than the level initially in 5e: It's tricky. Everything which showed up in a PHB in a prior edition has some analogue in 5e. For example, Warlord elements can be found in Battlemaster fighter options. They don't intend to have as many mechanical options though. They are focusing on the most popular elements from prior editions. You won't get to replicate a Xeph Psion from day one for instance as Psionics will take a while for them to get to.
Video #6 Questions
Question about the Free Basic Rules and OGL and Piracy issues:
Mearls felt making a version of the game free is helpful in reducing piracy. But ultimately making people a fan of D&D first is the most likely way to get them to buy content later (though he thinks the analysis is different for indie creators). He doesn't view the job of WOTC to be law enforcement. Mearls engaged in piracy for video games in his youth, like Ultima III, which led him to buying Ultima V later. He did the same with Napster, and then eventually moved to buying in iTunes because the storage stays on the cloud and was always there even when you switch computers or platforms. Piracy in his mind is just people sharing stuff as introductory, and if the content is high enough quality, and the physical book is high enough quality in art and texture and such, people will buy it. The best way to counter piracy is to offer a service customers really want.
Similarly Mearls mentions with digital products, doing something which makes the features and game faster and usability so great you'd want to pay for that rather than a static version is the key. [Note: This predicts DNDBeyond]
The basic game is intended to lower the barrier to try the game. A one-shot for a friend to try the game is accessible if they can just download those rules and sit down and play, or like at a convention. The goal is to make it easier for people to join us in our completely awesome hobby.
Question about the secret to a really good process: Iteration is part of it, and iterating with a purpose. Trust with the people you're working with is also highly important. Finally, having clear goals and vision early on so people know what they are working towards and not working at cross processes. Knowing your goals doesn't mean knowing your end product. Just that you know what you're constantly working towards philosophical objectives.
Questions: Which mechanics were difficult to say no to? Mearls says he really liked using dice instead of static bonuses, like plus 1d4 instead of +2, but nobody agreed with him. Some class mechanics, like the charge action, were kept out for good reasons. Like it undermined fast turns. Flanking was similar. There was a lot of argument about bonus actions and it was originally rejected by Mearls and took 3 months of persuasion and convinced Mearls it was his idea (though it was not).
Playtesting the DMG: The focus was what would make the DMs job easier. Lists of treasure were roughly done on just asking experienced DMs what they thought would be appropriate for a level. Playtest DMs sometimes drove the optional rules in DM, with requesting optional tools to add. Adding some additional styles of play which slightly varied from the core styles. DMG was the hardest book to do. They couldn't really do iterative testing the same way.
Questions on Monster races? They wanted to get away from penalties. Considering a lot of options on how to handle that. Not sure about the savage species route. Ability score penalties were not really dinging characters - if you had a negative charisma dwarf, you typically just wouldn't play a sorcerer and dumping charisma was not meaningful.
Question on contracting out Mayfair games: Mearls says don't want to dilute their focus. They don't tend to want to do other RPGs, unless they were adapting 5e to something else, because they are a big fan of slowly improving things over time with iteration and you lose that if you start from scratch with a whole new thing.
Question about the number of books people like to acquire: Mearls repeated people tend to buy the core books, and then four expansion books over a decade. Their mindset changed to much more user experience and looking at a big picture for the game. When looking at a new product they have to ask what it brings to the table. They are focused on cultural milestones and ways to make a new book an event and not just the next thing for the next month. Get people talk about D&D, even if they don't plan to buy something. They do a product because they can tell a story about that product that makes sense and exiting and is engaging. For example "The biggest dungeon ever published for D&D" and here is the book for the DM, and here is the book for the players with character options and cool lore. Things with backgrounds specific to the locations and setting. Things which can ground a character in that campaign.
They are being more story and adventure focused. By expansions people tend to think of mechanics and crunchy, and so maybe they will focus on just four big mechanics focused expansions with a lot of other books on story and adventure. So they're not always dumping a giant book of new mechanics out there.
Video #7 Questions
Focus on not just selling to people who are already playing the game by making sure marketing and advertising is comprehensible to someone who has never played.
For new character classes, which they have not done so far, they'd first ask "What's the hook that's going to make a person want to play that." There is no magic single formula for making a new class it's very focused on who that class would appeal to and providing mechanics which serves those expectations. Their big question is, is this a sub-type of an existing class or a truly new type of class. Most will be a new type of sub class.
Question which seems to be about how D&D is a very maker culture and a marketplace for...3d printed figures? Hard to hear the lengthy question but Hasbro has some agreement in place to acquire something along that lines maybe but nothing to announce. [This could be a hint on what is coming with the VTT. They could add you make your own PC digital figure, or a marketplace to buy your own digital figure.]
Monster CR Question: Effectively what they do is break it down to a monsters offensive and defensive strength considering accuracy, and then features and special attacks and legendary actions all feed into it. How long will this monster live is a question asked before adjusting. And then baseline numbers for how difficult a monster is derived from an average player numbers. They then compare that CR 1 baseline to that monster's offense and defense. It's a very complicated Excel spreadsheet.
Question about Rich Burlew's campaign that was never published: Some of his setting ended up in Eberron, like the undead elves. They did buy all three winning settings from the setting contest.
And that concludes this 7th and final video summary.
Last edited:
