Worlds of Design: Playtest Your Games

It’s one thing to write for RPGs and another to write RPG rules. And one of those differences is playtesting.

chess-4264325_960_720.png

Picture courtesy of Pixabay.

That’s Fast!​

There are role-playing game writers who can churn out over 2,000 words a day. That’s three “Worlds of Design” columns a day, including editing and finishing. I have never had anything like that kind of facility, not even when I use voice recognition rather than type! The average novel is 90,000 to 100,000 words long; my book about game design (Game Design: How to Create Video and Tabletop Games, Start to Finish) is 100,000 words; the (Protestant) Bible’s New Testament is 181,000+ words. So 2,000 words a day amounts to a book every two months or so.

How can you playtest much of this material when you’re producing that amount per day? You can’t. While some of the 2,000 words a day may be setting material that is mostly color rather than actual rules, some will be rules for add-ons to games, or game rules themselves.

Admittedly, sometimes I wrote without testing, back in the 70s and 80s when I was writing a lot for the RPG magazines. But because I wasn’t trying to produce vast amounts of material, I could playtest much of what I wrote in actual play. A reason why I don’t write RPG rules add-ons/changes now is that I use playtest time for my board games, not RPGs.

Why Playtest?​

Games are active, not passive, unlike many other individual arts such as painting or sculpture. Interaction with passive arts is minimal, and rarely enables the viewer to change the art. Good games change (are different) every time they are played, changed by the players to a greater or lesser extent.

We playtest to help avoid the kinds of changes that we (designers) deem undesirable. We may encounter wildly different playing styles, preferences far different from our own, when we playtest rules. A big mistake of novice game designers is assuming that everyone likes the same things the designer does. One person’s feast is another person’s garbage.

If you’ve ever GMed the same (non-storytelling) adventure for different groups, you know the vast difference there can be from one group to another.

Playtesting and Other Media​

In any individual art where someone or occasionally a group of people creates a work, they could decide to test it with an audience for acceptance to see what the audience thought, but most such works, music, sculpture, paintings, plays, novels, and so forth are not tested. This may be partly tradition, but in some cases it's because it's too difficult to change the work. For example, sculptures are hard to change.

Until recently, films were rarely tested with audiences before release, but now it's become easier to alter a completed film, especially with digital editing. The filmmakers make a film with several endings, or try different things in other parts of the film, then they'll test it with audiences to see what the audience likes or doesn't like. I think it's also common now for novelists to ask a small group of people to read their books before submission. Yet the consumer cannot change the work, it must be changed by the creator.

Contrast this with games. Games are intended to change as the player(s) experience them. A game that always went the same way would not be popular at all. People expect games to go differently each time they play. (Contrast with puzzles, which often are the same when successfully solved.)

There are now some modern artworks that are intended to change with input from the consumer. In other words, where the consumer, the user, participates, but this is a relatively recent development.

Game designers have much less control over their work, so the designer playtests the game to try to gain a form of control over how it is used by the players. And that control ensures that the constraints they've established in the game do work ... or don't result in dynamics that are undesirable.

We now have many video games that are released only partly completed so they can be modified and expanded afterward. Even the electronic version of my boardgame Britannia is going through Early Access on STEAM. Further, some massive multi-player online games (MMOs) and other online games are intended to be changed over time in the light of play. There's no pretense on launch that they won't change.

Playtesting vs. Revision​

Playtesting is different from revision. Most authors revise their work, though I know of a single exception. Isaac Asimov was a famous author of over 300 books, some fiction but most of them fact. His most well-known works are the science-fiction Foundation series. On a dare he once wrote a short story and sold it without revision—but he was Isaac Asimov. Anybody publishing an anthology or magazine would be happy to include an Isaac Asimov story even if it wasn’t good.

Revision, as opposed to playtesting, is changing things based on actual experience. If you don’t playtest with others, then that experience must be your experience only. It's what you do when you're solo testing a game that you've created—more than normal revision, but less than full playtesting. Playtesting is collecting experiences of others, then revising your game. I don't play in playtests of my own games because I want to see other people's experiences, I don’t want my experiences to influence the testers’.

I revise “Worlds of Design” columns a lot, but the “playtesting” comes from my wife (not a gamer these days) reading them [Ed Note: …and his editor!].

You can try to be like a typical creator of manual art and make a game without reference to how others experience it, but for games that can result in a poor product.

Your Turn: What playtesting experiences have you had where something needed to be drastically changed?
 
Last edited by a moderator:

log in or register to remove this ad

Lewis Pulsipher

Lewis Pulsipher

Dragon, White Dwarf, Fiend Folio

Jraynack

Explorer
Great article. Playtesting is essential to game design, though hard with smaller companies with much less reach. However, that does not mean it is impossible. There are a lot of game design Discords (NerdLAB for one), subreddits, and forums, as well as Tabletop Simulator to help broaden your playtest beyond the local area. As a small publisher for TTRPGs, short, but encompassing adventures, along with pre-generated characters can help discover design flaws (or even hidden gems) when played with several groups, even at the local level. So reach out to your local game community and don't be afraid to venture in unchartered territory to playtest your game.
 


Open playtests are a great tool. The larger the sample the more bugs are found.

The hard part is what some people consider bugs others consider features.
The much bigger issue is that many "open playtests" are not actually interested in doing any testing. They're just hype trains or (even worse) confirmation bias generators.

Actual, serious playtesting requires work, and most importantly, it requires unbiased data collection and actual data analysis. Many public playtests (I'm looking at you, WotC and Paizo) are absolutely LOADED with push polling, leading questions, biased answer options, and an inability to give meaningful feedback. Further, there's an incredibly frustrating tendency to both overreact and be completely blase about the results. Things that get slightly too much criticism but that one isn't strongly committed to will be obliterated and replaced, sometimes repeatedly, leading to a never-ending cycle of half-baked ideas or a failure to actually generate something new to test. Things that get lots of criticism but to which the team has a major ideological commitment will endure long past the point at which they should have been canned, dragging down the whole process.

The D&D Next playtest spent more than half of its time either dithering (failing to ditch unwise concepts a year or more after they had demonstrably proved untenable and widely disliked, e.g. proficiency dice) or vacillating wildly (the failure to settle on a Fighter concept until like two or three packets before the end of the playtest). Ideas should get more than one pass unless they are deeply unpopular, but should not get seven passes in the hope that maybe this version will be liked. And again, this isn't just Wizards; Paizo did exactly the same thing with many PF1e playtests, most notably the Gunslinger (a very badly-designed class where public playtesters gave useful, targeted feedback and were not only ignored but sometimes BANNED for doing so).

Playtesting is incredibly important. If WotC had rigorously playtested the ratio of rests per day that players actually use, rather than only what they wanted players to use, we literally might not be getting 5.5e now. They would have done a little more work initially to reap GREAT benefits later. Likewise 4e's stealth rules and Skill Challenge rules (which were half-baked at best early on), or pretty much ALL of 3e after about level 8 (hence the prevalence of "E6" rules.)
 

aramis erak

Legend
I'll note that running your own design should NOT be counted as playtesting. You'll fill in the gaps in what you wrote, where others might ask.
I didn't realize this until my game was in someone else's hands, and he had 30 questions in session 1.
Which lead to a revision....
 

TheSword

Legend
Playtesting is incredibly important. If WotC had rigorously playtested the ratio of rests per day that players actually use, rather than only what they wanted players to use, we literally might not be getting 5.5e now. They would have done a little more work initially to reap GREAT benefits later. Likewise 4e's stealth rules and Skill Challenge rules (which were half-baked at best early on), or pretty much ALL of 3e after about level 8 (hence the prevalence of "E6" rules.)

There is a price to be paid for too much play testing though, just as there is for too little. The problem is those systems did have play testing that delayed the release, and no doubt they could could have playtested endlessly and never released anything. Eventually a line is drawn and folks say this product is good enough to go out.

Perfection is the enemy of Progress - Churchill
The best is the enemy of the good - Voltaire
Better a diamond with a flaw, than a pebble without - Confucius

Calling WotC and Paizo’s testing of 1e and 5e a hype train is particularly unreasonable considering how it clearly shaped both of those products and cemented some progressive and popular ideas. It sounds like you just didn’t agree with the decisions taken on the back of it. Gunslinger was a popular addition as far as I could see, particularly for something so controversial as gunpowder.

Sometimes it’s better to release products into the wild and see what happens than study them in a lab. Particularly if are open to further changes based on live testing.

Believe me, 10 years on we would be getting some version of 5.5 whatever it turned out to be. From the looks of it, very similar to the existing system. WotC did get it right and did reap GREAT benefits from it. You sound a bit like the person looking at the Cistene Chapel’s ceiling and saying it would have been better on a wall so you didn’t have to crick your neck. As turning your head to the heavens wasn’t the whole point. The fact that 5e has lasted 10 years tells me that they pretty much got it right. Funnily enough as a result of play testing.
 
Last edited:

There is a price to be paid for too much play testing though, just as there is for too little. The problem is those systems did have play testing that delayed the release, and no doubt they could could have playtested endlessly and never released anything. Eventually a line is drawn and folks say this product is good enough to go out.

Perfection is the enemy of Progress - Churchill
The best is the enemy of the good - Voltaire
Better a diamond with a flaw, than a pebble without - Confucius

Calling WotC and Paizo’s testing of 1e and 5e a hype train is particularly unreasonable considering how it clearly shaped both of those products and cemented some progressive and popular ideas. It sounds like you just didn’t agree with the decisions taken on the back of it. gunslinger was a popular addition as far as I could see, particularly for something so controversial.

Sometimes it’s better to release products into the wild and see what happens than study them in a lab. Particularly if are open to further changes based on live testing.

Believe me, we would be getting some version of 5.5 whatever it turns out to be. From the looks of it, very similar to the existing system. That’s suggests to me that they pretty much got it right.
Well, the amount of time they spent on the public playtest would have been adequate, if they hadn't (a) held onto things like Expertise Dice and Specialties way longer than they should have, and (b) instantly scrapped other things, like the playtest Sorcerer and Warlock, never to be seen again until publication.

Two to three years is a solid amount of time to playtest and write a good game. WotC just wasted easily half that time either dithering or throwing the baby out with the bathwater.
 

dragoner

solisrpg.com
Play tests still need to be curated though. When I ran a T5 game in the play test forum, one thing I noticed were people always complaining it wasn't Classic Traveller, something it never was going to be. Though at the same time, I suggested it get broken into 3 300 page books vs one 650 page book; also it should have a "Streamlined T5" as a player's guide, or rules light version. Though the main motivator for the game has now passed, and all my observations are probably gone.

Now I think with the current stuff I am releasing, I will keep as close to the Cepheus Engine SRD rules as possible, or write my own rules, which I have done about halfway, just needing spacecraft, and world generation rules, mostly. Part of what I see now days is that people make some pronouncement of a game having sine qua non elements that make it that game, without having looked around to see a lot of newer games have the same thing.
 


Yeah its shame 5e was so unsuccessful, maybe 6e will be better :)
Popularity and quality are often unrelated things.

The most popular MMO in the world is still World of Warcraft (not by much, but still.) The most popular restaurant in the world, and indeed in most countries where franchises are a thing, is McDonald's. "Coke" is literally one of the few pan-human words because of its market share worldwide. The most popular beers are Bud, Bud Lite, Miller Lite, etc., accounting for something like a quarter of all beer sold worldwide.

Or, consider Prego. Before they introduced Extra Chunky and became market leader, they were losing to Ragu, even though on several levels Prego was a superior spaghetti sauce (higher quality ingredients, better texture/consistency, more tomato flavor). Adding Extra Chunky to their lineup didn't change the quality of either product, but it made Prego market leader.

Further, just because a system is popular does not mean it can't be flawed. 3rd edition was popular. It's also one of the most flawed editions ever written, having forced not one but two standard-bearers, even its great champion Paizo, to admit that it was too broken to keep developing for. Pathfinder 1e remains one of the most popular non-D&D5e games on Roll20, more popular than its much better-playtested successor.

"X succeeded, therefore it can't have issues" is a ridiculous fallacy. But, if you have any arguments you'd like to field that aren't fallacious, I would be happy to engage. Dismissing an opponent's position as necessarily false solely because they used a fallacious argument is, itself, a fallacy (with the terrible, meta name "the fallacy fallacy.")
 

Arilyn

Hero
I think 5e was more market research than playtest. The playtest packets often varied wildly, with the very first one being no more than a sketch of an idea. We never even saw ranger or bard until 5e came out.

Nothing wrong with market research but a good playtest is tightly focused, and given to groups who will really run the game through the wringer with knowledgeable, critical expertise.
 

aramis erak

Legend
Play tests still need to be curated though. When I ran a T5 game in the play test forum, one thing I noticed were people always complaining it wasn't Classic Traveller, something it never was going to be. Though at the same time, I suggested it get broken into 3 300 page books vs one 650 page book; also it should have a "Streamlined T5" as a player's guide, or rules light version. Though the main motivator for the game has now passed, and all my observations are probably gone.
Last I checked, the T5 playtest forum still exists, it's just only board staff and Marc can see it. Hell, the Legacy of the Aldenata playtest is still in the COTI database... but only staff and Marc can see it. Marc occasionally does access the T5 one.
Last I checked, T5 is now in a 3 volume set... T5 Traveller5 Core Rules 3-Book Set - Game Designers' Workshop (GDW) | Mega Traveller | Traveller 4 (T4) | Traveller: The New Era | Traveller (FASA/GameLords/JG) | Classic Traveller | DriveThruRPG.com
 

dragoner

solisrpg.com
Last I checked, the T5 playtest forum still exists, it's just only board staff and Marc can see it. Hell, the Legacy of the Aldenata playtest is still in the COTI database... but only staff and Marc can see it. Marc occasionally does access the T5 one.
Last I checked, T5 is now in a 3 volume set... T5 Traveller5 Core Rules 3-Book Set - Game Designers' Workshop (GDW) | Mega Traveller | Traveller 4 (T4) | Traveller: The New Era | Traveller (FASA/GameLords/JG) | Classic Traveller | DriveThruRPG.com
It was read only last I looked, still is, interesting with the new software, one can see how many threads people posted in. I was player in a game, not more than a couple of years ago? There is still some activity in the T5 forum, seems Robject has taken over.
 

aramis erak

Legend
It was read only last I looked, still is, interesting with the new software, one can see how many threads people posted in. I was player in a game, not more than a couple of years ago? There is still some activity in the T5 forum, seems Robject has taken over.
I'm not certain if he realizes only He, Marc, Thom, McPerth, Cryton, and I can post there... but it's supposed to be read only.
 

dragoner

solisrpg.com
I'm not certain if he realizes only He, Marc, Thom, McPerth, Cryton, and I can post there... but it's supposed to be read only.
T5 in general or only the play test is supposed to be read only? Looking at it just now, I am surprised at how long ago all that was, it closed on April, 2013, that is nine years ago.
 





Related Articles

Visit Our Sponsor

Level Up!

An Advertisement

Advertisement4

Top