Do You Think Open Playtests Improve a Game's Development?

[Reynard]

Open playtesting has become common in the TTRPG industry. I think it started in earnest with Pathfinder 1E's playtest, but I might have missed something earlier.

Anyway, I am not convinced that open playtests are actually beneficial to design and development. Especially when those playtests are rolled out slowly and in pieces (like the 5E playtests) I think that the community responses can smother changes and mechanics before people see them in a larger context or over a longer play time. I also think that most gamers don't know what they want and aren't very good game designers, so listening them has no guarantee of producing better results.

What do you think?

 

[Joshua Randall]

No, because people are haphazard and do not actually “test” anything. They play the game and give random stream-of-consciousness feedback.

 

[dbolack]

Open commentary from indirectly solicited vectors come with all manner of new perspectives but also hidden landmines and missing perspective.

 

[dbolack]

No, because people are haphazard and do not actually “test” anything. They play the game and give random stream-of-consciousness feedback.

That's really not that distinct from a random Con demo.

 

[Reynard]

Open commentary from indirectly solicited vectors come with all manner of new perspectives but also hidden landmines and missing perspective.

That's really not that distinct from a random Con demo.

I honestly can't tell whether you are saying you are pro or anti open playtesting.

 

[Joshua Randall]

That's really not that distinct from a random Con demo.

Umm… right. That’s exactly why open play”test”ing is useless. Because you’d get the same “results” (such as they are) from a random con demo. So just, like, do that.

Nowadays I guess you’d do it online.

 

[Joshua Randall]

To put on my UX hat:

Open playtesting (misnomer, but we’ll use it) conflates generative research with formative/summative research. By the time you have rules to test, you are way past the generative stage, the open ended stage where you take in a zillion possibilities that you will later synthesize down to a few designs.

By the time you have rules to test, you are at the formative/summative stage, where you should be, literally, testing the effectiveness of your designs.

I have never seen an open playtest actually do that formative/summative research properly.

 

[GMMichael]

I think closed playtests are just fine.

 

[DragonLancer]

My only experience here was with last year's Daggerheart open playtest. I loved the original release and my group gave feedback. Each subsequent playtest update removed or changed elements that I liked and it spoilt some of my enjoyment. I included that in the feedback but more and more I felt that I was perhaps not the right consumer of the direction they seemed to take the product, to the point that I dropped the playtest to rejoin D&D.

It might sounds odd but I think if I never partook of the open playtest and just bought/played the final product it might have been different. The initial playtest gave me different expectations that I would not have had just picking up a final product.

Maybe some players are better at open playtests than others. Maybe it's better to have a publishers vision from a closed playtest than to get hundreds of thousands of random potential players conflicting responses.

 

[Reynard]

My only experience here was with last year's Daggerheart open playtest. I loved the original release and my group gave feedback. Each subsequent playtest update removed or changed elements that I liked and it spoilt some of my enjoyment. I included that in the feedback but more and more I felt that I was perhaps not the right consumer of the direction they seemed to take the product, to the point that I dropped the playtest to rejoin D&D.

It might sounds odd but I think if I never partook of the open playtest and just bought/played the final product it might have been different. The initial playtest gave me different expectations that I would not have had just picking up a final product.

Maybe some players are better at open playtests than others. Maybe it's better to have a publishers vision from a closed playtest than to get hundreds of thousands of random potential players conflicting responses.

I feel similarly about the 2024 5E playtest, in that every time it looked liek they were going to do something daring, i got excited. then they pulled back and I got deflated.