Playtesting: As I was driving home last night, I thought of a couple other playtest-related things. First, back in that ENWorld thread, Sean Reynolds asks how big a list of external playtesters for 4e would be.
First, you'd think I'd have that list. But I don't. I've got some of it, Chris Tulach down in RPGAtopia has more, and Andy Collins has some names, too. We've had lots of different playtests with lots of different groups, and especially early on, keeping track of it all was catch-as-catch-can. Playtesting is something we got better at as we went along.
But because results pass through the wind tunnel that is my in-box, I might be the best person in the world to make an educated guess at the size of the list.
[Noonan takes long break to sift through playtest reports.]
To keep things reasonable, I'll count only people who knew they were playtesting a future edition of the game--playtesters for Book of Nine Swords, Star Wars Saga Edition, and Monster Manual V don't count. (Whether they should count is a different question, and an interesting one.) And I'm going to assume a table size of six for a few playtests where we had only the DM report in (which would make more sense if you knew what the test was). Doing some back-of-the-3x5-car d math, I think we're at something like 400 so far. Give me a plus/minus of 50 on that.
Well, hmm. That specific number is going to beg comparisons to 3rd edition playtesting. So it's important to deliver a caveat. The 4e playtesting methodology is a lot different than the 3e playtesting methodology was. Much of the 3e playtesting boiled down to "Here's the manuscript...tell us what you think." In some ways, that's a great playtest. Your testers are motivated (hell, they're frickin' elated!) and you get some great snapshots into ongoing play. Testers are often really good at big-picture issues that come up in an ongoing campaign--after all, they're staring at the future of their chosen hobby, so they ask some very pointed questions.
The other cool thing about that playtest methodology is you see what people play when you give them the whole sandbox. If they collectively gravitate toward certain play styles and certain mechanical elements (classes, races, etc.), well, you just gained insight into a fundamental question of RPG design: "Will anyone care about this thing I'm making up?" But it's a double-edged sword. In an open playtest like that, you don't control what people actually test. If few people choose druids, for example, any issues with that class may remain unseen until you're done with playtesting. If few people engage in high-level play, then you don't get much feedback on the high-level experience.
The other problem with that playtest technique: Whoa, Nellie, is it duplicative. Mind-numbingly duplicative. The individual tables of testers never see it, but I'm pretty sure Kim Mohan's eyes glazed over after the 200th consecutive complaint about an early version of Weapon Focus. From the perspective of the individual tester, of course you'd want to spend a lot of time telling Wizards how Weapon Focus should work. But you don't--and can't--realize that it's wasted effort given that dozens of other tables are going to write the same thing, and poor Kim is going to spend hours reading every one of them. And our playtesters are volunteers with limited time, so that time writing the Weapon Focus treatise could have been spent checking out the druid's animal companion, for example.
We've done some of that "Here's the manuscript...tell us what you think" testing too. But in general, our testing is a lot more focused. We often want these specific PCs against those specific monsters. We want to see n builds of a x-level ranger. Even some really micro-level stuff--we watch you play to see how long it takes you to add up the dice on that damage roll, how long it took you to find that rule, and (one of my favorite questions) where you looked for it first.
So maybe at the end of the day, the number of external eyeballs on 4e will be something similar to the number of 3e eyeballs. But it's fundamentally an apples-and-oranges thing. Our 4e testing is using a fundamentally different appraoch. This makes the argument less fun, which is too bad.
And I'll end with my big-picture take on 4e playtesting: Nobody in the tabletop hobby industry does a better job playtesting than Wizards does. But I wish we playtested more. I think our games could be even better.
One of the things on my plate right now is an effort to do something about that--a pretty aggressive playtesting plan for the RPG products that will follow the core books. (Pro Tip: Join the RPGA and start participating.)
