Polling, Open Playtesting and Game Design

Hussar · Jun 15, 2012

I was reading the latest Dragon's Eye View article on the WOTC site (for those who don't read it, it's the art column) and something caught my eye. The poll discussing what version of the Owlbear people liked. We had a pretty clear winner:

WOTC Site said:
Which owlbear do you prefer? (From last week's article)
Grizzowl 01 ------- 45.2%
Screecher 03 ------- 16.6%
Track 2 owlbear ------ 11.4%
Gorrilowl 04 ------- 8.8%
Tallgrizz 02 ------- 7.6%
Oldgrump 06 ------- 4.8%
Original owlbear ------ 4.2%
Longbeak 05 ------- 1.5%
Total -------- 100.0%

Grizzowl01 (which was my choice to be honest) is the clear winner here. It's got everyone else beat hands down.

But, there's a problem. It is the odds on favourite, true, but, it doesn't have the majority of people on board. If we choose this image and go forward, we're actually losing over half of the fans of the owlbear.

So, therein lies the problem. If you are designing a game, what do you do? Do you go with the odds on favourite, even knowing that you've just ignored over half the gaming population? Do you go back to the drawing board and try to find an image that does appeal to over half? Is half good enough? How about 2/3rds?

I can really see WOTC being stuck behind a rock and a hard place. They are trying to appeal to the broadest number of gamers possible, but, in doing so, leave themselves open to the "Gnome Effect" to use Mike Mearl's term.

Say you have a group of 6 players. From the above poll, 3 of the six players aren't happy with your decision. If half the group doesn't like something, it's pretty likely that they're not going to play your game.

I'm not sure what can be done about this. I suppose the open playtest is a good start. Take the feedback, loop it back around and try to find as much common ground as possible. Like the song says, two out of three ain't bad, so that might not be a terrible benchmark.

Just a meandering thought.

Herremann the Wise · Jun 15, 2012

The thing that you're not taking into account here (and that the poll is not judging either) is that the winning entry may have been the second choice for a lot of those polled. It most certainly does not mean that 54.8% of those polls didn't like it. Perhaps it is only "hated" by a small percentage and so going with that option is not as perilous as you envisage.

Perhaps a better poll (or follow-up poll now that the results are known) is which images do you find acceptable representations of the owlbear. This may lead to the conclusion that the 45.2% option is well supported (even if not everyone's favourite) or it could show that another option is more neutrally accepted. My point is that you need to be very careful with statistics as its very easy to misrepresent what a set of unclear statistics is not saying. [I am a mathematician by the way

].

Best Regards
Herremann the Wise

Quickleaf · Jun 15, 2012

Hussar said:
So, therein lies the problem. If you are designing a game, what do you do? Do you go with the odds on favourite, even knowing that you've just ignored over half the gaming population? Do you go back to the drawing board and try to find an image that does appeal to over half? Is half good enough? How about 2/3rds?

What I've really enjoyed about Jon Schindehette's articles is that he brings a strong creative vision to D&D's art. And he does it in a fairly open-minded way, not being wedded to any one of his ideas. I know when I'm juggling competing interests at work, or at the gaming table, and I need a tie-breaker, I always go back to my creative vision.

I'm not sure I see a creative vision for 5e's game design yet. Certainly Jon is making strides in the art department.

I can really see WOTC being stuck behind a rock and a hard place. They are trying to appeal to the broadest number of gamers possible, but, in doing so, leave themselves open to the "Gnome Effect" to use Mike Mearl's term.

I've read the article where Mearls discusses the "Gnome Effect" (Dungeons & Dragons Roleplaying Game Official Home Page - Article (What's With the Polls?)) and I'm not sure I grasp what he's saying. Either it's painfully obvious (change the game and not only do individual players react but also gaming groups) or it's going over my head.

Campbell · Jun 15, 2012

Considering feedback is important, but at the end of the day game designers, artists, graphic designers, editors, etc. need to man up and make the tough decisions about a variety of trade-offs and focus on making a good product that offers utility above and beyond the options consumers currently have available.

I think one element that commonly gets lost in open play tests and general polling is the fact that you need to discern if it comes from a potential customer. Can you provide them with utility above and beyond the substitute goods available to them without alienating too many of your other potential customers? The other element to consider is that what people claim to want and what they actually want can be quite different. What matters in the end is their economic actions. Do they buy the core rules? Can you market supplemental material to them? Is the design talent you have available to you able to serve their needs without too much cost? Will they bring new players into the network?

Those are all hard questions. If you could design a highly profitable game through polling and design by committee there would be a lot more highly profitable games. This isn't an easy business. Potential consumers have highly specific tastes and a wealth of substitute goods available to them.

Blood & Bones · Jun 15, 2012

Hussar said:
But, there's a problem. It is the odds on favourite, true, but, it doesn't have the majority of people on board. If we choose this image and go forward, we're actually losing over half of the fans of the owlbear.

So, therein lies the problem. If you are designing a game, what do you do? Do you go with the odds on favourite, even knowing that you've just ignored over half the gaming population? Do you go back to the drawing board and try to find an image that does appeal to over half? Is half good enough? How about 2/3rds?

<snip>

I'm not sure what can be done about this. I suppose the open playtest is a good start. Take the feedback, loop it back around and try to find as much common ground as possible. Like the song says, two out of three ain't bad, so that might not be a terrible benchmark.

I think the answer is to have players rank their choices rather than pick one. This way, if nobody agrees with their number one choice, but everybody agrees on their number two choice, they get something they can live with rather than simply the choice that has the most votes (which most likely will not have a plurality vote).

Does that make sense? I feel that I might be explaining it too obscurely.

Ratskinner · Jun 15, 2012

Herremann the Wise said:
The thing that you're not taking into account here (and that the poll is not judging either) is that the winning entry may have been the second choice for a lot of those polled. It most certainly does not mean that 54.8% of those polls didn't like it. Perhaps it is only "hated" by a small percentage and so going with that option is not as perilous as you envisage.

Perhaps a better poll (or follow-up poll now that the results are known) is which images do you find acceptable representations of the owlbear. This may lead to the conclusion that the 45.2% option is well supported (even if not everyone's favourite) or it could show that another option is more neutrally accepted. My point is that you need to be very careful with statistics as its very easy to misrepresent what a set of unclear statistics is not saying. [I am a mathematician by the way ].

Best Regards
Herremann the Wise

I agree, and would only add that a "...and how much do you care?" question would help clarify things as well. Plenty of people play this game (if forum discussions are any evidence) while holding fairly strong objections to more central things than the presented image of one monster.

Blood & Bones · Jun 15, 2012

Quickleaf said:
I've read the article where Mearls discusses the "Gnome Effect" (Dungeons & Dragons Roleplaying Game Official Home Page - Article (What's With the Polls?)) and I'm not sure I grasp what he's saying. Either it's painfully obvious (change the game and not only do individual players react but also gaming groups) or it's going over my head.

I think the point is that simple one vote polls are deceptive. You can have 45% of people prefer that gnomes not exist in the core rules. You have 55% pointing to other races that they would like to see kicked out of the PHB - Half-Orcs, Half-Elves, Halflings, Dwarves, Elves, and Humans in decending order.

Now, since you can choose only one, you of course choose the race you most dislike. Now, lets say that of the people who voted for gnomes, most of them would also like to see Halflings axed. Now they REALLY hate gnomes, but halflings don't warm their heart either. And lets say halflings are the second choice for just about everyone who voted on the poll... You could easily have 89% of people against halflings - more than those who voted against gnomes - and not know it!

Another way of thinking of it is to say that only 10% of players are willing to play a gnome. Well, you think to yourself, that's 10% you can afford to lose. However, what you don't know is how much of that 10% are DM's or influential players within their group. If those players feel disenfranchised and they have pull as to what is played, how likely are those groups to actually play D&D 5th Edition? We might be looking at a good 33% of groups switching to Pathfinder - because Pathfinder has Gnomes, and we want to keep Johnny Coolbeans happy. EVERYONE likes Johnny Coolbeans.

That's 33% of the market that WotC cannot afford to lose over a stinky gnome! And all they had to do to keep that demographic is plug the big-nosed buggers into the original release of the PHB!

Does that make sense?

Hussar · Jun 15, 2012

Blood & Bones said:
I think the answer is to have players rank their choices rather than pick one. This way, if nobody agrees with their number one choice, but everybody agrees on their number two choice, they get something they can live with rather than simply the choice that has the most votes (which most likely will not have a plurality vote).

Does that make sense? I feel that I might be explaining it too obscurely.

Yes, that makes perfect sense. Ok, like I said, I was just musing. It makes sense to springboard from the simple poll into more complicated ones.

GX.Sigma · Jun 15, 2012

Quickleaf said:
I've read the article where Mearls discusses the "Gnome Effect" (Dungeons & Dragons Roleplaying Game Official Home Page - Article (What's With the Polls?)) and I'm not sure I grasp what he's saying. Either it's painfully obvious (change the game and not only do individual players react but also gaming groups) or it's going over my head.

The point of that article is that the polls on the WotC articles are not a form of market research. Due to a variety of reasons, including the "gnome effect", these polls are completely useless as a means of gathering information. They have a whole department (which interfaces with third party market research companies) to do that sort of stuff, and they're very good at what they do.