Do You Think Open Playtests Improve a Game's Development?

[Reynard]

In the case of 5.5, given that the core mechanics were largely unchanged, and what was being changed were modifications to classes, monster, spells and the like, why wouldn’t piecemeal work? It’s not like the mechanics were unfamiliar to anyone. Perhaps the biggest change were the weapon properties as a new subsystem within the game, but the game was not completely rebuilt from the ground up.

This was asked a bunch of times during the 2024 playtest, and all I can say is that I don't buy it. there were some changes that would have impacted play as a whole, but we did not have a whole game to test. Most of thsoe were pulled back on.

I will reiterate my position just to make it clear: I don't think that the fandom willing to answer surveys are necessarily good at providing design insight, and I don't think that testing games less than holistically does much good.

 

[TiQuinn]

This was asked a bunch of times during the 2024 playtest, and all I can say is that I don't buy it. there were some changes that would have impacted play as a whole, but we did not have a whole game to test. Most of thsoe were pulled back on.

I will reiterate my position just to make it clear: I don't think that the fandom willing to answer surveys are necessarily good at providing design insight, and I don't think that testing games less than holistically does much good.

Again, I disagree. You’re making a blanket statement that seems to be based on your personal dissatisfaction with one particular game (and playtest). Again, I don’t see how that makes for a strong argument regarding the value of playtests.

 

[Leatherhead]

The D&D Next playtest wasn't actually a playtest. It was more like an open marketing feedback survey.

 

[Reynard]

Again, I disagree. You’re making a blanket statement that seems to be based on your personal dissatisfaction with one particular game (and playtest). Again, I don’t see how that makes for a strong argument regarding the value of playtests.

I'm not basing it on one playtest, I am basing it on watching different playtests of different styles and their outcomes.

But you are right in that I am making a general statement that I don't think open playtests work. I made that clear from the OP.

You feel differently. That's cool.

 

[BrokenTwin]

I mean, for point of comparison, video game playtesting is notoriously difficult to do right, and they have numerous advantages over pen & paper games (larger sample sizes, the ability to embed metric-gathering into the game itself).
Do I think open play testing works best for ensuring maximum market saturation? Yes. Do I think it enables the developers to produce work that aligns closest to their vision? No. I've seen it play out a lot in general media, and while there are some success stories, there's so many more pieces of art that were completely ruined by focus group testing and too many competing interests.

 

[Umbran]

Do I think it enables the developers to produce work that aligns closest to their vision? No. I've seen it play out a lot in general media, and while there are some success stories, there's so many more pieces of art that were completely ruined by focus group testing and too many competing interests.

Well, let us be clear - user-research isn't about creating art. It is about creating a PRODUCT. The research is about aligning that product with the needs and desires of customers.

 

[Campbell]

Open playtests are decent at providing acceptance testing (whether or not users will accept or value a given feature). They are poor at providing systems testing results (how components interact with one another). Targeted internal testing or with outside groups you can depend on to follow testing instructions are still necessary for testing how a game actually plays rather than if its features will get initial acceptance from its audience.

 

[BrokenTwin]

Well, let us be clear - user-research isn't about creating art. It is about creating a PRODUCT. The research is about aligning that product with the needs and desires of customers.

Oh yes, that is the point I was (kind of) trying to make. When people are answering the question "do open playtests improve a game?", you first need to confirm what metric you're judging "improve" by. Otherwise you'll have people arguing right past each other (which has already happened a few times in this thread, from my perspective).

 

[dbolack]

Some playtests are done only for hype that they generate.
Some are done solely to catch typos.
Some are done to check acceptability of new rules.
Some are done to check the whole game.
At least one was primarily to get player input on the setting...

Sometimes purely a Vibe check...

 

[loverdrive]

My experience with playtesting (granted, I'm making a videogame) is that the only useful thing I get from playtesters is footage, especially with experienced playtesters who can narrate their thoughts, lets play style.

The thing they write afterwards is a complete and utter garbage, and I'm more than sure I give similarly useless feedback when asked — turns out it's hard to analyze the game while playing it! Who would've thought.