Do You Think Open Playtests Improve a Game's Development?

[payn]

It's complicated. I think the impression from playtest feedback is important, but a designer needs to be able to separate experience from function. For example, if folks are saying they dont like something, is it becasue the mechanics do not work, or is it because they dont like how it feels in play. There are two different corrections there. One is obvious, tweak the mechanics until they work correctly. The other is find out how to get the right feel. Perhaps its the presentation of the mechanics that is causing confusion?

I also like the olive branch to the community with an open playtest. I know that im one of thousands and my voice will probably be drowned out. I do appreciate the effort for a peak under the hood though. So, It doesnt bother me if the playtest isnt as sound as it should be. At least its enaging the fanbase and generating excitment. I think there is value in that.

I think feel is as important as innovation. Im self-aware enough to know I probably care a bit more about design than the average gamer. That doesnt give my opinions any more objectivity though. Folks dont seem to be able to separate those things all that well. YMMV

 

[jian]

The only time I took part in an open playtest was with 5E (D&D Next) and it was a mixed result. I really liked a lot of things they tried (manoeuvre dice, the martial sorcerer, etc) but then they removed all of those things and the final game felt really uninspired by comparison. However, when we played it, it was a pretty decent D&D game. Overall, if we’d just bought the final product and never taken part in the playtest I think we’d have been happier.

 

[payn]

The only time I took part in an open playtest was with 5E (D&D Next) and it was a mixed result. I really liked a lot of things they tried (manoeuvre dice, the martial sorcerer, etc) but then they removed all of those things and the final game felt really uninspired by comparison. However, when we played it, it was a pretty decent D&D game. Overall, if we’d just bought the final product and never taken part in the playtest I think we’d have been happier.

There is that rub for sure. I was a big PF1 player and took part in the PF2 playtest. I do think there was a lot more change based on feedback then what I experienced with 5E. However, I disagreed with almost all the changes. I ended up not adopting PF2, but im pretty happy that a lot of folks enjoy it quite a bit.

The cat is out of the bag really. I think at this point having done both open and closed playtesting, I cant say I prefer one or the other. I have seen both generate questionable results, and positive results based on the feedback. Being naturally interested in design, im always going to question things, and always think some things are better and others are worse. Open or closed wont change that. YMMV.

 

[Umbran]

What do you think?

I think there's a very valuable skillset in getting feedback from customers and really understanding it.

Yes, people are not themselves great designers. No, they don't know what they want. But, if you work with them, know how to ask the right questions, and actively listen to the answers, you can find out what does and doesn't work for them. You can determine what their problems actually are, and help fix them.

 

[Joshua Randall]

You have a very limited perspective on value that can be gained. Or perhaps are only interested in a very limited subset of values. That's fair. But that isn't what everyone is looking for and not always what is found.

Do enlighten me, then.

I run user research for a living.

 

[Umbran]

You have a very limited perspective on value that can be gained.

I think you are incorrect. I mean, how can you speak to his perspective on value when he wasn't speaking about value at all!

He's talking about doing the research in what a professional in the field would call a proper manner.

To analogize - no statistician or polling professional would look at the polling tech we have here on EN World and call it statistically relevant. That doesn't mean you can't learn anything from them, but it does put some significant limits on their utility.

 

[Umbran]

It's complicated. I think the impression from playtest feedback is important, but a designer needs to be able to separate experience from function. For example, if folks are saying they dont like something, is it becasue the mechanics do not work, or is it because they dont like how it feels in play.

... or because it is just different, or...

This is why real user research is difficult, and often expensive.

 

[Joshua Randall]

I think there's a very valuable skillset in getting feedback from customers and really understanding it.

Yes, people are not themselves great designers. No, they don't know what they want. But, if you work with them, know how to ask the right questions, and actively listen to the answers, you can find out what does and doesn't work for them. You can determine what their problems actually are, and help fix them.

this is an excellent and succinct description of the value of UX / user research, in general

 

[TiQuinn]

I believe it was in the 50th anniversary Gen Con video for 5e, Mike Mearls said that the feedback to DnDNext was invaluable to them. The results just maybe didn’t manifest in the way playtesters thought it would in many cases, but judging by the result, it was beneficial.

 

[Reynard]

I believe it was in the 50th anniversary Gen Con video for 5e, Mike Mearls said that the feedback to DnDNext was invaluable to them. The results just maybe didn’t manifest in the way playtesters thought it would in many cases, but judging by the result, it was beneficial.

It is possible, but there is no way to know for sure. After all, I think it is pretty broadly accepted that 5E's success is NOT a function of its system, but of a convergence of many factors -- not least being media presence (Stranger Things) and streaming (Critical Role). I am sure it helped that 5E is both easy to learn and easy to stream, but I don't know that any of the elements considered in the playtest and abandoned would have resulted in a less playable or less streamable game.

In other words, you can't just say "5E was successful ergo 5E's final design is best."