Do You Think Open Playtests Improve a Game's Development?

[mamba]

We know both Stranger Things and Critical Role increased interest in D&D. What we can't know is if it "stuck" because of 5E system choices.

Yes, but we also know that it already grew before them, ‘all’ they did was accelerate a process that was already ongoing. So another thing we do not really know is how much difference to the growth rate and ultimate player numbers they actually made either.

Here is a chart from Alphastream about the Bookscan numbers, I added CR and ST, first season for both. Stranger Things dropped all episodes on one day.

Estimating D&D’s Revenue | Alphastream

The BookScan data gives us a potential window into estimating D&D’s overall revenue, especially when combined with previous clues. If we can estimate the percentage BookScan might be of D&#038…

alphastream.org

 

[dbolack]

Do enlighten me, then.
I run user research for a living.

That doesn't seem likely to be productive.

 

[Reynard]

Yes, but we also know that it already grew before them, ‘all’ they did was accelerate a process that was already ongoing. So another thing we do not really know is how much difference to the growth rate and ultimate player numbers they actually made either.

Here is a chart from Alphastream about the Bookscan numbers, I added CR and ST, first season for both. Stranger Things dropped all episodes on one day.

View attachment 394292

Estimating D&D’s Revenue | Alphastream

The BookScan data gives us a potential window into estimating D&D’s overall revenue, especially when combined with previous clues. If we can estimate the percentage BookScan might be of D&#038…

alphastream.org

The initial numbers are a lot of returning D&D players. We know that's true.

 

[Reynard]

Okay, but then explain what you mean by “I am not convinced that open playtests are actually beneficial to design and development” if the actual designers are seeing value in it?

Ultimately, are you really criticizing the value of playtests or are you criticizing the choices the designers make based on the information they receive and act upon based on those playtests?

I am saying that I don't think listening to the community in an open playtest necessarily makes a better game, and in fact can make for a worse game because designers are distracted and influenced by the most invested fans who are opinionated but not necessarily right.

5E turned out well. That's awesome. What we can't say is if it turned out better than it might have, if vocal minorities hadn't shouted down some ideas. The same with 2024.

 

[dragoner]

Yes, as long as you know how to filter the results. Mostly it is best to see if things are working how you want them to vs trying to kitchen sink everything to satisfy people.

 

[TiQuinn]

I am saying that I don't think listening to the community in an open playtest necessarily makes a better game, and in fact can make for a worse game because designers are distracted and influenced by the most invested fans who are opinionated but not necessarily right.

5E turned out well. That's awesome. What we can't say is if it turned out better than it might have, if vocal minorities hadn't shouted down some ideas. The same with 2024.

Okay, I think that really overestimates the overall influence that invested fans have while underestimating the intelligence and experience of those actually working on the game. I can’t think of a case where having less knowledge about your customers using your product resulted in a better result, in any industry really.

 

[aramis erak]

Open playtesting has become common in the TTRPG industry. I think it started in earnest with Pathfinder 1E's playtest, but I might have missed something earlier.

Anyway, I am not convinced that open playtests are actually beneficial to design and development. Especially when those playtests are rolled out slowly and in pieces (like the 5E playtests) I think that the community responses can smother changes and mechanics before people see them in a larger context or over a longer play time. I also think that most gamers don't know what they want and aren't very good game designers, so listening them has no guarantee of producing better results.

What do you think?

I think an open playtest followed by a closed playtest can tighten a game up considerably. I've seen this done with Dune and Star Trek Adventures...

But playtesting only is good if the process is valued by the designer.

Some playtests are done only for hype that they generate.
Some are done solely to catch typos.
Some are done to check acceptability of new rules.
Some are done to check the whole game.
At least one was primarily to get player input on the setting...

Those each get better games out... but the best playtests are about all but the hype.
think Pathfinder 1e.
Think Starfinder.
Both put everything up for playtester discussion. And they listened. And are, according to people I know actually playing them, better than the playtest materials.

 

[payn]

I am saying that I don't think listening to the community in an open playtest necessarily makes a better game, and in fact can make for a worse game because designers are distracted and influenced by the most invested fans who are opinionated but not necessarily right.

5E turned out well. That's awesome. What we can't say is if it turned out better than it might have, if vocal minorities hadn't shouted down some ideas. The same with 2024.

How is better measured? It seems popular approval is out. Making lots of money is out. I’m not sure of an objective tool because even the feel of the game impacts enjoyment, which is entirely subjective. How would you determine better?

 

[Reynard]

Okay, I think that really overestimates the overall influence that invested fans have while underestimating the intelligence and experience of those actually working on the game. I can’t think of a case where having less knowledge about your customers using your product resulted in a better result, in any industry really.

Maybe I wasn't clear.

I don't believe that the fans that respond to open playtesting a) necessarily represent the broad average of players, and b) I don't think those fans motivated to answer are necessarily motivated to make the game better and instead are motivated to make thegame stay the way they like it.

I think designers should be trusted to designa game.

If you are going to do a playtest, it certainly should not be piecemeal and in brief intervals. you should put out a wholly playable version of the game and let people play it as if they would the complete game, and take feedback on that.

 

[billd91]

5E turned out well. That's awesome. What we can't say is if it turned out better than it might have, if vocal minoritie shouted down some ideas. The same with 2024.

Well, that’s one, largely snotty, spin on it. But what’s the difference between a vocal minority shouting something down and an insufficiently large majority embracing it? Rude spin is all. I mean what IS it about people ascribing negative personal characteristics behind decisions they don’t happen to like? Does it make you feel better about yourself and your own preferences?