Playtesting: Scott called me about this thread on ENWorld. Interesting stuff throughout. I can shed at least a little light on what’s going on with playtesting right now.
We’ve playtested a lot of different ways. When I plan a playtest wave, I think of it in terms of altitude—how far up the observer is when looking down at the game. High-altitude playtesting is an ongoing campaign, where you’ll see characters (and players, for that matter) evolve over time. You have mid-altitude playtesting, which might be a single adventure or an attenuated campaign. And you have low-altitude playtesting, which is a single encounter repeated ad nauseum, or a sequence of escalating/deescala ting variations (“let’s try it at 3rd level…now 4th…now 5th…”).
So altitude is one axis. You also have different things you’re looking for within that altitude band. You might want to see mechanical interactions—how the numbers are matching up (specifically or generally). You might care about speed of play—both in “game world time” and “real world time.” You might be seeing which adjudication of a situation works best at the table.
And you also test the away-from-table stuff such as character generation, encounter/adventure /campaign design, and the learning process (how long before someone groks opportunity attacks, for example).
No playtest technique can capture all of that. If you had to pick a single technique, you’d probably go with ongoing campaigns for playtesting purpose, because they have the salient advantage of most closely mimicking actual play. (In particular, one of the things that drives me crazy is that most people play more aggressively with pregenerated characters than with “their” characters. It’s totally understandable, but you sometimes get weird results that way.) But ongoing campaigns a) take a long time; and b) do a poor job of actually capturing small data points, because everyone’s focused on the overall game.
Scott mentioned the “Prison Break” playtests we did a wave or two ago. This was a simple 10-room dungeon with four versions: 2nd, 4th, 6th, and 8th level. You start at the back end of the dungeon and your goal is to see daylight again, basically. We provided the characters, and we provided the specific encounters. So we got to see exactly how dozens of tables would react to more-or-less the same circumstances. This turned out to be a pretty good low- to mid-altitude playtest.
Here’s what the player questionnaires looked like (the DMs got a different questionnaire). I’m going to [Blank out!] some stuff because it refers to specific mechanical elements that haven’t been revealed yet—and in one case, we’ve had a significant change in terminology:
=====
1) What's your character class and level?
2) How many XP did you earn in this session?
3, repeated) Repeat for each encounter: You’re going to give us a numeric summary of your “output.” Note what your damage output was per round. Just write down the number.
• If you missed, put down a "0."
• If you didn't attack, put down an "X."
• If you attack when it's not your turn (like an oppo attack), put the damage in parentheses, and put a zero in parentheses if you miss during an oppo or something like that.
• If you got KOed or killed, put down a "—."
• And if you attack multiple targets in the same round (with an area attack or by [Go Bears!], for example), use slashes to separate the output.
• If you used any [Van Halen Rules!] or [Hi Mom!], write that the end of the string.
Ask your DM for the room number. Use commas to separate the rounds. At the end of the session, a 2nd-level wizard might have a string for each encounter that looks something like:
Wizard 2 vs. Room 1: 14/7, 8, 0, 10, X, 6, 0. Used 1 [Hi Mom!].
Wizard 2 vs. Room 2: 9, 15, 11 (0), 0, 21. Used 2 [Hi Mom!]s and [Van Halen Rules!].
4) Did you notice any rules problems or unclear stuff while playing? What was it, and how did you resolve it at the table?
5) Did you discuss quitting for the day and resting? If so, how many at the table wanted to quit and how many wanted to continue? What did the group ultimately decide? (Repeat if the discussion happened more than once.)
6) Put yourself in the 3e mindset for a second. Did your experience feel like the D&D you're used to? Why or why not?
7) What other character at the table impressed you the most (race and class)? What did they do that was so cool?
8) Was there a PC that didn't seem to be pulling his/her weight? Keep in mind that this isn't a reflection on the player--just the character (race and class is sufficient). And what was the trouble?
9) What was the coolest monster you faced, and why? Oh, and the one you liked the least, whether for balance/flavor/tabl e play reasons--tell us about that one too.
10) Anything else the designers/developer s should know about?
======
That’s the questionnaire. I can tell you, it feels really good to paste all those output strings into a single spreadsheet, then start torturing them until the conclusions come out. One example: I’d always wondered how many rounds of in-combat downtime exist for a PC or a monster. In other words, how often does a PC simply double-move or get out a key and open a door, or something similar that’s useful but doesn’t directly, mechanically contribute to the outcome of the encounter. The “Prison Break” wave was big enough that I think we’ve got the answer—for the covered levels, at any rate.
But the qualitative stuff is just as important. The thoughtful answers we received to question #6 were particularly useful, as I recall.
