Who Playtested This Anyway?
  • Who Playtested This Anyway?


    The first playtests of Dungeons & Dragons were by Gary Gygax, his kids, and his friends. The industry has evolved considerably since then and playtesting along with it. A new playtesting methodology was borrowed from software development, and it's likely to influence how game companies produce products in the future.


    Photo by Ian Gonzalez on Unsplash

    The Early Playtests

    Gary Gygax, co-creator of Dungeons & Dragons, originally playtested early drafts of D&D with his kids according to David Ewalt:

    The first people to play it were Gygax’s eleven-year-old son, Ernie, and nine-year-old daughter, Elise. Gygax had created a counterpart to Arneson’s Blackmoor, which he called Castle Greyhawk, and designed a single level of its dungeons; one night after dinner, he invited the kids to roll up characters and start exploring. Ernie created a wizard and named him Tenser— an anagram for his full name, Ernest. 3 Elise played a cleric called Ahlissa. They wrote down the details of their characters on index cards and entered the dungeon. In the very first room, they discovered and defeated a nest of scorpions; in the second, they fought a gang of kobolds— short subterranean lizard-men. They also found their first treasure, a chest full of copper coins, but it was too heavy to carry. The two adventurers pressed on until nine o’clock, when the Dungeon Master put them to bed. Fatherly duties completed, Gygax returned to his office and designed another level of the dungeons.

    The RPG industry didn't exist back then as we know it today, so the limited scope of his early playtests were understandable. He expanded that scope over time to include his friends and colleagues. Things changed once the RPG industry matured. Gygax would often publish early rules in Dragon Magazine as a form of playtest release. In later editions of D&D, this relationship between magazine and game became more formal:

    During its second Wizards of the Coast run, the Dragon magazine staff was aligned with the D&D R&D staff. This allowed the magazine to integrate more closely with the actual D&D game than ever before. This began in Dragon 360 with a new “Design & Development” column—which didn’t just preview D&D 4e, but also explained the reasoning behind many of the design decisions, offering a level of interaction with the D&D creators that had never been seen before. Starting with Dragon 365 (July 2008), readers could also playtest upcoming D&D rules—the first of which were drawn from the Eberron Player’s Guide (2009).

    That alignment happened around Fourth Edition, which was when D&D started borrowing elements of video game development for its roll-out, including playtesting.

    The Industry Grows Up

    Role-playing games are bound by a basic premise that "anything can be attempted," and thus while there are rules that can be playtested, there are a limitless number of potential unforeseen consequences in a game where players have full agency over their character. Video games are more constrained, although Massive Multi-player Online Role-Playing Games (MMORPGs) have become increasingly complex because of the interactions between masses of players. To that end, video games use open alpha- and beta-stage testing. These playtests release the unfinished game to a limited audience so that the developers can observe and record feedback. Minecraft was an important example of how this works:

    An example of a video game that made extensive use of open playtesting is Minecraft, which was made available for purchase in its pre-alpha stages. This both helped to financially support the game and provide feedback and bug testing during its early stages. Playtesting began even before the game features included multiplayer or the ability to save games. Mojang continues to make use of playtesting with Minecraft through weekly development releases, allowing players to experiment with unfinished additions to the game and provide feedback on them.

    It's worth noting that playtesting is used as both a form of mass feedback on the game and a marketing opportunity to generate revenue, essentially asking players to pay for the privilege of making the game better in exchange for early access. In fact, the practice has become so commonplace that Game Informer changed its policy to review video games as soon as they begin charging customers instead of waiting until the game is considered "finished." For a particularly egregious example, see Fortnite Battle Royale, which has been in "Early Access" since 2017!

    This same approach to playtesting was evident when Wizards of the Coast sold copies of the "Wizards Presents" series for Fourth Edition, which included early previews of the game before it saw the light of day. The playtest for the Fifth Edition, perhaps in reaction to the struggles of Fourth Edition, took on a whole new dimension with an open call that included over 120,000 playtesters. Mike Mearls, senior manager of research and design for D&D at the time, explained the strategy to Fortune:

    In some ways we're just catching up to the resources you have in digital games, where you can look at server logs and see what characters are people making, what abilities are they using. It’s kind of us embracing a more modern approach to game design. Especially with something like Dungeons & Dragons, that has such a clear identity and such a huge fan base, it lets us really stay in touch with what people are experiencing. Honestly, without this data and without the feedback we would be designing a much different game.

    Just as they did with Fourth Edition, Wizards released a playtest version for sale in Ghosts of Dragonspear Castle. Since then, playtesting has become increasingly a public, data-driven affair that has been used by other large RPG publishers, including Paizo:

    In 2008, Paizo launched an unprecedented public playtest aimed at updating the third edition rules to make them more fun, easier to learn, and better able to support thrilling fantasy adventures. More than 40,000 gamers just like you joined in the fun by playtesting the new Pathfinder RPG rules and providing feedback, and the rest is gaming history. Now, 10 years later, it's time to put the lessons of the last decade to use and evolve the game once again. It's time for Pathfinder Second Edition!

    Like Wizards, Paizo published print versions of the playtest for purchase:

    PDF editions of the Pathfinder Playtest Rulebook, the softcover Pathfinder Playtest Adventure, and the Pathfinder Playtest Flip-Mat Multi-Pack will be available for FREE right here on paizo.com starting August 2. The print edition of the Playtest Rulebook is available in softcover ($29.99), hardcover ($44.99), and deluxe hardcover ($59.99) editions. The print edition of the Playtest Adventure has a suggested retail price of $24.99, and the Flip-Mat Multi-Pack is priced at $24.99.

    There are downsides to this transparent approach to game development as White Wolf discovered in a 2017 playtest for the Fifth edition of Vampire. A character named Amelina was described as preying on children -- and the fallout from that playtest (among other issues) led to some serious changes to the company as a result.

    The engagement with the audience in shaping role-playing games doesn't end with playtesting though.

    Night of the Living Games

    Mearls explained how feedback would be ongoing as a "living game":

    The biggest change affects how we make updates to the game going forward. In the past, we relied on forums, summaries of issues from customer service, and our own experiences with the game to guide the changes we made. Though this approach uncovers parts of the game that people are having issues with, it does a poor job of assessing the magnitude of those issues. The public playtest showed us that we need to cast a much wider net to create a clear picture of what’s going on. To that end, you can expect to see annual surveys that work much like the ones we used to guide the development of fifth edition D&D. These surveys will not only allow us to identify trouble spots in the game, but we can use them to look at how attitudes change over time. By comparing one year’s results to the next, we can gain a sense of how the game is changing.

    Plenty of smaller publishers playtest too, of course. Chaosium is notable for including playtest notes in some of its products, so game masters can see how a scenario evolved. Fantasy Flight Games has its own playtesting program, to name a few.

    Increasingly, playtesting is no longer limited to groups within a company, but rather open to all who are willing to put in the work of playing the game and sharing their feedback. It's not uncommon to hear the refrain, "who playtested this?" at the table.

    These days, it's most likely one of us.

    Mike "Talien" Tresca is a freelance game columnist, author, communicator, and a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to http://amazon.com. You can follow him at Patreon.
    Comments 31 Comments
    1. DQDesign's Avatar
      DQDesign -
      when playtesting lasts more than a reasonable amount of time is just a way like another to artificially reduce enterprise running risk.
    1. Lylandra's Avatar
      Lylandra -
      not necessarily. Playtesting done right can include multiple iterative loops of alternatives and feedback. At worst (or best?) you have to break down and rebuild a lot if you realize that you are about to upset your playerbase.

      Now video games have the benefit of easy data collection and fast, frequent play. TTRPG systems are only played every so often even by avid players and data collection is almost exclusively done via optional surveys. So they naturally need more time to be thoroughly playtested.

      5e, imo, is a success in terms of playtesting and player feedback, even if it still has its drawbacks (for me, personally, as an individual player). We will see if Paizo can repeat this with their PF2e.
    1. Henry's Avatar
      Henry -
      There are downsides to this transparent approach to game development as White Wolf discovered in a 2017 playtest for the Fifth edition of Vampire....
      This was actually an excellent example of how some companies make the mistake that open playtesting takes the place of QA, rather than supplementing QA....

      Plenty of smaller publishers playtest too, of course. Chaosium is notable for including playtest notes in some of its products, so game masters can see how a scenario evolved.
      I actually wish more companies did this, a "liner notes" if you will on the headspace of the designer, so that some of the more ambiguous items in a rule set might be clarified. I admit it can be tough to know if a particular component is going to be troublesome, however.

      In computing, the Free and Open Source Software community (FOSS) is a great example of how "playtesting" my be done well. The problem is that "bug reports" are a different concept in game playtesting, and a lot of people don't understand what is helpful feedback (then again, in the open source community, the same often applies... ) Plus, game designers are not as keen as FOSS developers are on accepting help from others on "fixing" a specific problem -- the only difference there is that there are plenty of coders who are competent to design code fixes, whereas there are a lot of armchair game designers (such as myself) who don't understand all the ramifications of a proposed change, and some even take rejection of a suggested fix as if it were an attack on their character.

      I am all for more open playtesting, however -- its benefits outweigh the problems, as we've seen through both 5th edition D&D, Pathfinder first edition, and (hopefully) even Pathfinder Second edition. (Hey, they killed resonance, so we know open playtesting works! )
    1. SMHWorlds's Avatar
      SMHWorlds -
      You do have to be rigorous about who you have playtest and how you incorporate that into your design, especially as an Indie dev. I am still perfecting that aspect of design.
    1. EthanSental's Avatar
      EthanSental -
      The paizo pathfinder 1e and the D&D 5e playtests seem to have gone pretty well turning out great games.
    1. ParanoydStyle's Avatar
      ParanoydStyle -
      It does not sit well with me when game companies (analog or digital) charge people to be part of a playtest. I mean, it probably won't ever stop until fans refuse to pay, and I don't know how to get that to happen short of a miracle, but... yeah, paying money to help a company balance its incomplete game has always rubbed me the wrong way. I take more issue with it on things like Steam early access than here.
    1. R_Chance's Avatar
      R_Chance -
      Quote Originally Posted by ParanoydStyle View Post
      It does not sit well with me when game companies (analog or digital) charge people to be part of a playtest. I mean, it probably won't ever stop until fans refuse to pay, and I don't know how to get that to happen short of a miracle, but... yeah, paying money to help a company balance its incomplete game has always rubbed me the wrong way. I take more issue with it on things like Steam early access than here.
      In both cases (Paizo 2008 and current, and WotC 5E) paying for anything was an option. The PDFs were free.

      I bought the original PF playtest and a hardcover of the new PF2E. I played PF 1E in playtest and final form, the 2E was curiosity, to see where the game was going. Not my cup of tea. I playtested 5E without purchasing a hardcopy, I just downloaded the PDFs. I think the simpler iterative system of WotC made it easier to test (and print out) the game without hardcopy. I prefer PF 1E to 2E and 5E over both. Although the pull of OSR systems is strong...
    1. Ben Wilkinson's Avatar
      Ben Wilkinson -
      I like the 3.x playtest method. "Don't and hope for the best."
    1. Geekoid's Avatar
      Geekoid -
      The play testing used in 4e was one of the worse things to happen to D&D.
    1. Blue's Avatar
      Blue -
      Quote Originally Posted by Henry View Post
      I actually wish more companies did this, a "liner notes" if you will on the headspace of the designer, so that some of the more ambiguous items in a rule set might be clarified. I admit it can be tough to know if a particular component is going to be troublesome, however.
      13th Age does this. Plenty of sidebars abtu why the designers did something one way, and the effects of tweaking and changing things. Also when the designers disagreed and both points of view. Really helps you understand the "why" of the rules, as well as make it much more hackable.
    1. R Karan's Avatar
      R Karan -
      Putting together a good survey can be hard and will affect the answers you get. Are there any examples (from D&D or PF) available online?
    1. Greenstone.Walker's Avatar
      Greenstone.Walker -
      Quote Originally Posted by DQDesign View Post
      when playtesting lasts more than a reasonable amount of time is just a way like another to artificially reduce enterprise running risk.
      I'd take out the word "artificially".
      It is about actually reducing risk.
    1. Shasarak's Avatar
      Shasarak -
      Quote Originally Posted by ParanoydStyle View Post
      It does not sit well with me when game companies (analog or digital) charge people to be part of a playtest. I mean, it probably won't ever stop until fans refuse to pay, and I don't know how to get that to happen short of a miracle, but... yeah, paying money to help a company balance its incomplete game has always rubbed me the wrong way. I take more issue with it on things like Steam early access than here.
      I dont mind paying to playtest something that I like. How else are you going to get a chance to change things from the inside?
    1. Grimkrieg's Avatar
      Grimkrieg -
      This 'improved' playtesting methodology still let Armor of Agathys through...
    1. Gadget -
      Quote Originally Posted by Grimkrieg View Post
      This 'improved' playtesting methodology still let Armor of Agathys through...
      ?? I was unaware of any major complaints about Armor of Agathys?
    1. lewpuls's Avatar
      lewpuls -
      I'm going to have to disagree. Playtesting, often widespread playtesting, was common in board games long before that became common in the video game industry. And when the video game industry playtested, it tested for bugs, for ways that the softrware didn't work as intended, rather than for "fun testing", whether the software (when it worked as intended) was actually engaging or "fun" to play.Testing in video games was by the people within the company, many of whom were intimately involved in the game. That wasn't (and isn't) the way to do "fun testing", with people too close to the game.The tabletop game industry taught the video game industry to do fun testing. I remember talking about this in articles and comments for years; now it's not usually a problem, though you still get video game people who think that metrics is all you need for good playtesting.Of course the open betas and early releases are (often) a way to get people to pay for a product that isn't finished, in effect to get people to pay to do your playtesting! That's quite common.I remember reading once that one of Epic's Gears of War games was playtested 40,000 hours [sic]. That's practically impossible for tabletop games, other than big releases like D&D5 or Pathfinder.
    1. dragoner's Avatar
      dragoner -
      Something to remember about playtesting is that it is only as good as those willing to listen, having participated in a few, I have seen issues that have come up in play ignored for various reasons.
    1. Bobble's Avatar
      Bobble -
      Quote Originally Posted by talien View Post
      That alignment happened around Fourth Edition, which was when D&D started borrowing elements of video game development for its roll-out, including playtesting.
      And that produced a very craptastic, smelly cow patty. That almost destroyed the line.

      I don't think S/W testing procedures are needed. Play testing should be VERY late in the process. Your marketing work should tell you the basic data you need in order to design your game. The 4E crew could not conduct good market research to save their lives. If you cannot design, no amount of play testing is going to help you. DESIGNING the mechanics is NOT the job of testers; See Marc's disaster with Traveller.

      If you design well and get a good person to be your Editor you should just need a few rounds of limited playing testing to find the holes in the rules, streamline, clarify and polish. 99% of the job shroud be done BEFORE play testing.
    1. aramis erak's Avatar
      aramis erak -
      Quote Originally Posted by Bobble View Post
      And that produced a very craptastic, smelly cow patty.
      I disagree quite strongly with this. As do its thousands of fans.

      It plays well. Playtesting provided for a game that runs smoothly, provided you don't fight the system.

      It just doesn't feel at all like other versions...
      Quote Originally Posted by Bobble View Post
      That almost destroyed the line.
      This part is true, but not for the reason you imply.

      The combination of being so different, not being OGL, 3.X being OGL, and Paizo launching Pathfinder....

      PF also did a big public playtest... and made several changes from 3.5... but kept the base user experience, at least until they had their own playerbase.


      PF being the old game updated and D&D4E being a new game was a big chunk.

      PF continuing to be OGL and allowing 3PP supplements while D&D being closed content with very limited 3PP, was a second chunk.

      Playtesting for PF was as much advertising as genuine playtest. For many, like me, it was proof that it wasn't for me. For many others, it was proof that 3.X could be improved and still be familiar.

      Open playtesting is a form of advertisement, as well as a means of improving the game.

      It worked well for Mongoose, for Modiphius, and for Paizo. The paid open for Star Wars worked really well for FFG.
    1. Bobble's Avatar
      Bobble -
      Quote Originally Posted by aramis erak View Post
      I disagree quite strongly with this. As do its thousands of fans.
      I'm not referring to opinion, mine or others as that is subjective and cannot be right nor wrong. But, simply how the market reacted to the product. That is unarguable (sanely) based on sales and it knocking D&D out of first place in RPG history. Remember, the market is ALWAYS right.
    Comments Leave Comment