David Noonan's historical perspective on 3.0 (Update: Part III posted)

[Glyfair]

And I don't know what "average" playtesting times are, but it seems to me that this is a VERY short time-frame; playtesting packets are just going out now, and the book (PHB) is due out in less than a year, with "promo products" out considerably sooner that that.

Remember, they were playtesting semi-internally for quite a while (there was at least one example of a non-WotC employee in the playtest reports.

Really? You think?

I disagree; I tend to think you'd get MORE useful opinions than by telling the groups to "Test rules X, Y & Z." (I don't know how rperesentative of the D+D audience I am, but if I were playtesting, and given free rein, I would LOOK for things that could be broken, or "niche" things.)

With D&D yes. With a more limited rules set it could work that way. An RPG like D&D has far too many rules to give them all to a group and say "test these and tell me what you find in a week." Maybe you give them all the rules. However, you certainly say "test this & that with this group of characters." The next wave you say "we changed this and this, try these, and test these new areas as well."

It all comes down to how trustworthy (NDA-wise) you know your playtesters are when it comes to giving them everything. I wouldn't be surprised to see the experienced RPGA groups get more stuff to look through than the random D&D Insider members.

 

[ShadowDenizen]

Remember, they were playtesting semi-internally for quite a while (there was at least one example of a non-WotC employee in the playtest reports.

Out of curiosity, how long would you imagine "quite a while" is? (Six months? 1 year? Longer?)

And how many people participated in these tests?

And how many are true RPG'ers? (IIRC, there were mentions of "newbies" [people brought in from marketing] as some of the "internal playetesters.) Would you rather have an Erik Mona or James Jacobs, or someone who has never played the game before as a playtester?

(Though I can see the benefit to having new players in small numbers as playtesters, I would think you'd get the most insightful comments from people who KNOW the game, and what potential problems may exist.)

An RPG like D&D has far too many rules to give them all to a group and say "test these and tell me what you find in a week." Maybe you give them all the rules. However, you certainly say "test this & that with this group of characters." The next wave you say "we changed this and this, try these, and test these new areas as well."

I would think that (IMO) the logical thing to do is give the testers a chapter a week, and test those rules, until the while book is complete. (IE: "This week, we'll focus on spells. Everyone make a specialist caster.")

THen, on the second round, they could give "Updated" versions of each chapters.

***

HMMM.
I'm finding this subject fascinating for some reason; maybe we should start a "Playtesting" thread.

 

[WayneLigon]

This is an example of misidentifying the problem.

I think it's simply more of 'we want to make the game you want to play'. I'd prefer broader 'archtypes' than classes, but lots of people love classes and they seem to particularly love those. So, they stayed, along with the built-in assumptions and flavor.

 

[VirgilCaine]

Some of those pre-gen characters were really wimpy. You can find them in Enemies and Allies. The example Lidda didn't even take Weapon Finesse, despite being a halfling rogue! Instead, she tried to rely on Strength-boosting items.

I have to say, whoever came up with or whatever method was used to generate the sample 3.0 DMG PCs, needs to be booted from having anything to do with 4e. I want GOOD sample NPCs, not NPCs that look like they were made using random charts!

I started with 3rd edition, and even in my earliest days as a player, I hope I never made such grave mistakes in feat or magic item selection!

 

[Glyfair]

Out of curiosity, how long would you imagine "quite a while" is? (Six months? 1 year? Longer?)

They've said they were working on it for several years, so I wouldn't be surprised to hear at least a year with heavy playtesting since February.

And how many people participated in these tests?

Initially, I would guess one or two groups. They mentioned shutting down R&D for a month earlier this year for heavy playtesting.

And how many are true RPG'ers? (IIRC, there were mentions of "newbies" [people brought in from marketing] as some of the "internal playetesters.) Would you rather have an Erik Mona or James Jacobs, or someone who has never played the game before as a playtester?

As mentioned, everyone in R&D seems to have been involved. Most of the newbie comments seem to have been very recent (in fact, the one example started in the last week).

Don't underestimate the ability of "newbies" to test a game. A friend of mine used to work at a local computer game company. He pointed one of the best game testers was the secretary with little computer experience. She wasn't conditioned to video games like the programmers were, so often found stuff that wouldn't otherwise get touched.

I would think that (IMO) the logical thing to do is give the testers a chapter a week, and test those rules, until the while book is complete. (IE: "This week, we'll focus on spells. Everyone make a specialist caster.")

Now, I've never tested RPGs. However, the games I tested started with a limited amount of info (often very limited), with more added each week as the previous stuff was perfected (although there were always things needing tweaking throughout the process).

 

[Mustrum_Ridcully]

I think one of the problems with playtesting a new game system is that the players are simply so unfamiliar with it.
Imagine they would really need to digest all the rules first and than create new characters, and play them to level 30. That's obviously impossible in the given time frame. (You can't extend test times infinitely, because you need to sell the product before you run out of money )

So, the approach is probably: Let's build several sample characters at different levels, and provide the players with some encounters they have to "beat".

The players still have a lot of catching up to do, but you remove a lot of the initial build-up. And you can also improve the test coverage.

In software testing, unit-testing is a very typical part of testing. This means you don't test the whole application, you test small subsets (units) of it with test data - does this method/function/subroutine return the expected results with the given test data?
You will also later have some component/module testing (does this sets of methods/functions/subroutines work together as expected)
Even later you will make a system test and check if all works together.
Selecting test data is also important - you have two approaches - black box and white books. In case of black box testing, you only consider the interface specificiation of the testable unit (which gives you what type of data you can enter and what types of outputs you should expect). In case of white box testing, you look how the codes actually behaves dependend on input values and (warning: simplified) try to cover each possible path through the code.

On a D&D base, a single unit test is probably something like a skill or BAB. Quickly tested.
A Component Test might be test for a class or the spell system. More difficult to test, but at this point, play testers can't really help you much, as you can't "play" just a class.
A System Test would be the real play-testing.
Now, you get to choose your "test data" to input.
Black Box testing would probably be: "Take these characters and see how they fare in these encounters".
White Box testing would probably be: "How does Fighter Build 1 compare to Fighter Build 2 in this encounter (or maybe even: How does this Melee Warrior Build compare to this Spellcaster Build in this encounter?)". I am not sure how much White Box testing is actually done in playtesting - This is probably the point where you really test the fineprints of your system and find the hidden roads to unholy (Wotc Boards) Character Optimization. But there are so many possible builds and thus test cases that testing it all might take forever until you find something.
So, this might be the type of "bug" that the user finds out at home...

 

[Reynard]

I think one of the problems with playtesting a new game system is that the players are simply so unfamiliar with it.
Imagine they would really need to digest all the rules first and than create new characters, and play them to level 30. That's obviously impossible in the given time frame. (You can't extend test times infinitely, because you need to sell the product before you run out of money )

Alternatively, you could design the game in such a way that you can in fact sit down, create a character and get playing in under 30 minutes and let all the optional levels of detail and complexity come in as the game progresses. You could probably even do that ina single 64 or 128 page softbound book (maybe even in a big box with lots of rooms for minis, maps, etc...).

One of the things that is driving me batty about 4E is that on the one hand it seems to be being designed for people who aren't D&D/RPG players in mind, trying to pull in a different demographic, but on the other hand they are still going with the 3 giant books at a hige buy in method. They two don't make a lot of sense when put together.

If 4E is going to be a fast paced adventure game for people that enjoy CCGs, MMOs and CMGs then having a zillion options in giant hardbound books, requiring someone to "digest the rules" and play long term just to "get it" doesn't make any sense. If it is going to be an RPG in the sense that D&D has been for 30 years, it doesn't need to get MMO-ized the way they are going.

Pick one.

Or, don't, and give us 2 versions of the game. It worked fine in 1984, quite frankly.

 

[Mustrum_Ridcully]

Alternatively, you could design the game in such a way that you can in fact sit down, create a character and get playing in under 30 minutes and let all the optional levels of detail and complexity come in as the game progresses. You could probably even do that ina single 64 or 128 page softbound book (maybe even in a big box with lots of rooms for minis, maps, etc...).

One of the things that is driving me batty about 4E is that on the one hand it seems to be being designed for people who aren't D&D/RPG players in mind, trying to pull in a different demographic, but on the other hand they are still going with the 3 giant books at a hige buy in method. They two don't make a lot of sense when put together.

If you want to find the loop holes, you need to dig through the full rulebooks. You can start playing D&D by just looking at some few basics. But you will miss many flaws in this approach (except those flaws that make it difficult for new players to start a new game - which are also important)

I am not sure, but won't there be a adventure that will be released prior or together with the rulebook that will use the new rules, albeit only in the context of the adventure?
Anyway, new D&D editions certainly also always intended to get new players into the game, but part of the D&D concept seems also to be keeping them interested in the game. I guess the complexity of the books is just a result of this. There must be some rule foundation that allows all the zillion options to that you will expand your game.

If 4E is going to be a fast paced adventure game for people that enjoy CCGs, MMOs and CMGs then having a zillion options in giant hardbound books, requiring someone to "digest the rules" and play long term just to "get it" doesn't make any sense. If it is going to be an RPG in the sense that D&D has been for 30 years, it doesn't need to get MMO-ized the way they are going.

I think they are trying to combine the best in both worlds. Add those elements that we learn to like in MMOs or CRPG and so on, and keep those that make the D&D/Table Top RPG games unique and entertaining. They might fail. I am optimistic and think they won't. (Even if over time, we might find few weak spots that eventually warrant a D&D 5). But that's just me ...

Spoiler

and maybe some other D&D 4 fanboys

 

[Glyfair]

(You can't extend test times infinitely, because you need to sell the product before you run out of money )

In fact, when I see playtesting discussed online I notice that most threads contain someone who doesn't consider that playtesting costs money. Extend playtesting and you eventually will have to increase the price of the product.

First, you need a playtest coordinator position (which might be existing employees, but then they aren't doing their regular job during that time - which effectively costs you money). You have to have the existing data organized in format for the playtesters. You have to come up with questions for the playtesters to answer. You have to organize the data you receive to give to the designers for them to consider and address.

There are also a lot of admin duties. This group didn't get the data. Another group needs to make a playtester change. This group needs to quit. Don't even mention tracking down possible leaks.

 

[Mustrum_Ridcully]

In fact, when I see playtesting discussed online I notice that most threads contain someone who doesn't consider that playtesting costs money. Extend playtesting and you eventually will have to increase the price of the product.

First, you need a playtest coordinator position (which might be existing employees, but then they aren't doing their regular job during that time - which effectively costs you money). You have to have the existing data organized in format for the playtesters. You have to come up with questions for the playtesters to answer. You have to organize the data you receive to give to the designers for them to consider and address.

There are also a lot of admin duties. This group didn't get the data. Another group needs to make a playtester change. This group needs to quit. Don't even mention tracking down possible leaks.

What I wonder - do playtesters get something for their work? Are they paid, or do they just get free PHBs or something like that? That certainly isn't the case for public beta-testing in software, but that might be something different...