Polling, Open Playtesting and Game Design




+ Log in or register to post
Results 1 to 10 of 10
  1. #1
    Registered User
    The Grand Druid (Lvl 20)

    Hussar's Avatar

    Join Date
    Aug 2004
    Posts
    15,806

    Ignore Hussar

    Polling, Open Playtesting and Game Design

    I was reading the latest Dragon's Eye View article on the WOTC site (for those who don't read it, it's the art column) and something caught my eye. The poll discussing what version of the Owlbear people liked. We had a pretty clear winner:

    Quote Originally Posted by WOTC Site
    Which owlbear do you prefer? (From last week's article)
    Grizzowl 01 ------- 45.2%
    Screecher 03 ------- 16.6%
    Track 2 owlbear ------ 11.4%
    Gorrilowl 04 ------- 8.8%
    Tallgrizz 02 ------- 7.6%
    Oldgrump 06 ------- 4.8%
    Original owlbear ------ 4.2%
    Longbeak 05 ------- 1.5%
    Total -------- 100.0%
    Grizzowl01 (which was my choice to be honest) is the clear winner here. It's got everyone else beat hands down.

    But, there's a problem. It is the odds on favourite, true, but, it doesn't have the majority of people on board. If we choose this image and go forward, we're actually losing over half of the fans of the owlbear.

    So, therein lies the problem. If you are designing a game, what do you do? Do you go with the odds on favourite, even knowing that you've just ignored over half the gaming population? Do you go back to the drawing board and try to find an image that does appeal to over half? Is half good enough? How about 2/3rds?

    I can really see WOTC being stuck behind a rock and a hard place. They are trying to appeal to the broadest number of gamers possible, but, in doing so, leave themselves open to the "Gnome Effect" to use Mike Mearl's term.

    Say you have a group of 6 players. From the above poll, 3 of the six players aren't happy with your decision. If half the group doesn't like something, it's pretty likely that they're not going to play your game.

    I'm not sure what can be done about this. I suppose the open playtest is a good start. Take the feedback, loop it back around and try to find as much common ground as possible. Like the song says, two out of three ain't bad, so that might not be a terrible benchmark.

    Just a meandering thought.
    The rules don't give the DM their authority. The consent of the players does. - Mallus

 

  • #2
    Registered User
    Magsman (Lvl 14)

    Herremann the Wise's Avatar

    Join Date
    Apr 2003
    Location
    Sydney Australia
    Posts
    2,286
    Blog Entries
    4

    Ignore Herremann the Wise
    My communities:

    The thing that you're not taking into account here (and that the poll is not judging either) is that the winning entry may have been the second choice for a lot of those polled. It most certainly does not mean that 54.8% of those polls didn't like it. Perhaps it is only "hated" by a small percentage and so going with that option is not as perilous as you envisage.

    Perhaps a better poll (or follow-up poll now that the results are known) is which images do you find acceptable representations of the owlbear. This may lead to the conclusion that the 45.2% option is well supported (even if not everyone's favourite) or it could show that another option is more neutrally accepted. My point is that you need to be very careful with statistics as its very easy to misrepresent what a set of unclear statistics is not saying. [I am a mathematician by the way ].

    Best Regards
    Herremann the Wise
    Imagination is a quality given a man to compensate him for what he is not,
    and a sense of humor was provided to console him for what he is.

    Oscar Wilde

    He who is certain he knows the ending of things when he is only beginning them is either extremely wise or extremely foolish; no matter which is true, he is certainly an unhappy man, for he has put a knife in the heart of wonder.
    Tad Williams

  • #3
    Kung Fu Panda DM
    The Great Druid (Lvl 17)

    Quickleaf's Avatar

    Join Date
    Jun 2004
    Location
    Honolulu, HI
    Posts
    4,492
    Blog Entries
    5

    Ignore Quickleaf
    Quote Originally Posted by Hussar View Post
    So, therein lies the problem. If you are designing a game, what do you do? Do you go with the odds on favourite, even knowing that you've just ignored over half the gaming population? Do you go back to the drawing board and try to find an image that does appeal to over half? Is half good enough? How about 2/3rds?
    What I've really enjoyed about Jon Schindehette's articles is that he brings a strong creative vision to D&D's art. And he does it in a fairly open-minded way, not being wedded to any one of his ideas. I know when I'm juggling competing interests at work, or at the gaming table, and I need a tie-breaker, I always go back to my creative vision.

    I'm not sure I see a creative vision for 5e's game design yet. Certainly Jon is making strides in the art department.

    I can really see WOTC being stuck behind a rock and a hard place. They are trying to appeal to the broadest number of gamers possible, but, in doing so, leave themselves open to the "Gnome Effect" to use Mike Mearl's term.
    I've read the article where Mearls discusses the "Gnome Effect" (Dungeons & Dragons Roleplaying Game Official Home Page - Article (What's With the Polls?)) and I'm not sure I grasp what he's saying. Either it's painfully obvious (change the game and not only do individual players react but also gaming groups) or it's going over my head.
    Last edited by Quickleaf; Friday, 15th June, 2012 at 02:49 AM.

  • #4
    Registered User
    Defender (Lvl 8)

    Campbell's Avatar

    Join Date
    Feb 2004
    Location
    Denver, CO
    Posts
    1,507

    Ignore Campbell
    Considering feedback is important, but at the end of the day game designers, artists, graphic designers, editors, etc. need to man up and make the tough decisions about a variety of trade-offs and focus on making a good product that offers utility above and beyond the options consumers currently have available.

    I think one element that commonly gets lost in open play tests and general polling is the fact that you need to discern if it comes from a potential customer. Can you provide them with utility above and beyond the substitute goods available to them without alienating too many of your other potential customers? The other element to consider is that what people claim to want and what they actually want can be quite different. What matters in the end is their economic actions. Do they buy the core rules? Can you market supplemental material to them? Is the design talent you have available to you able to serve their needs without too much cost? Will they bring new players into the network?

    Those are all hard questions. If you could design a highly profitable game through polling and design by committee there would be a lot more highly profitable games. This isn't an easy business. Potential consumers have highly specific tastes and a wealth of substitute goods available to them.
    Storyteller 100%| Tactician 100 %| Butt Kicker 92%|Power Gamer 75%|Specialist 67%| Method Actor 58%| Casual Gamer 42%
    -Relaxed Intensity

  • #5
    Quote Originally Posted by Hussar View Post
    But, there's a problem. It is the odds on favourite, true, but, it doesn't have the majority of people on board. If we choose this image and go forward, we're actually losing over half of the fans of the owlbear.

    So, therein lies the problem. If you are designing a game, what do you do? Do you go with the odds on favourite, even knowing that you've just ignored over half the gaming population? Do you go back to the drawing board and try to find an image that does appeal to over half? Is half good enough? How about 2/3rds?

    <snip>

    I'm not sure what can be done about this. I suppose the open playtest is a good start. Take the feedback, loop it back around and try to find as much common ground as possible. Like the song says, two out of three ain't bad, so that might not be a terrible benchmark.
    I think the answer is to have players rank their choices rather than pick one. This way, if nobody agrees with their number one choice, but everybody agrees on their number two choice, they get something they can live with rather than simply the choice that has the most votes (which most likely will not have a plurality vote).

    Does that make sense? I feel that I might be explaining it too obscurely.

  • #6
    Registered User
    Myrmidon (Lvl 10)



    Join Date
    Feb 2012
    Location
    North Akron
    Posts
    1,606

    Ignore Ratskinner
    Quote Originally Posted by Herremann the Wise View Post
    The thing that you're not taking into account here (and that the poll is not judging either) is that the winning entry may have been the second choice for a lot of those polled. It most certainly does not mean that 54.8% of those polls didn't like it. Perhaps it is only "hated" by a small percentage and so going with that option is not as perilous as you envisage.

    Perhaps a better poll (or follow-up poll now that the results are known) is which images do you find acceptable representations of the owlbear. This may lead to the conclusion that the 45.2% option is well supported (even if not everyone's favourite) or it could show that another option is more neutrally accepted. My point is that you need to be very careful with statistics as its very easy to misrepresent what a set of unclear statistics is not saying. [I am a mathematician by the way ].

    Best Regards
    Herremann the Wise
    I agree, and would only add that a "...and how much do you care?" question would help clarify things as well. Plenty of people play this game (if forum discussions are any evidence) while holding fairly strong objections to more central things than the presented image of one monster.

  • #7
    Quote Originally Posted by Quickleaf View Post
    I've read the article where Mearls discusses the "Gnome Effect" (Dungeons & Dragons Roleplaying Game Official Home Page - Article (What's With the Polls?)) and I'm not sure I grasp what he's saying. Either it's painfully obvious (change the game and not only do individual players react but also gaming groups) or it's going over my head.
    I think the point is that simple one vote polls are deceptive. You can have 45% of people prefer that gnomes not exist in the core rules. You have 55% pointing to other races that they would like to see kicked out of the PHB - Half-Orcs, Half-Elves, Halflings, Dwarves, Elves, and Humans in decending order.

    Now, since you can choose only one, you of course choose the race you most dislike. Now, lets say that of the people who voted for gnomes, most of them would also like to see Halflings axed. Now they REALLY hate gnomes, but halflings don't warm their heart either. And lets say halflings are the second choice for just about everyone who voted on the poll... You could easily have 89% of people against halflings - more than those who voted against gnomes - and not know it!

    Another way of thinking of it is to say that only 10% of players are willing to play a gnome. Well, you think to yourself, that's 10% you can afford to lose. However, what you don't know is how much of that 10% are DM's or influential players within their group. If those players feel disenfranchised and they have pull as to what is played, how likely are those groups to actually play D&D 5th Edition? We might be looking at a good 33% of groups switching to Pathfinder - because Pathfinder has Gnomes, and we want to keep Johnny Coolbeans happy. EVERYONE likes Johnny Coolbeans.

    That's 33% of the market that WotC cannot afford to lose over a stinky gnome! And all they had to do to keep that demographic is plug the big-nosed buggers into the original release of the PHB!

    Does that make sense?

  • #8
    Registered User
    The Grand Druid (Lvl 20)

    Hussar's Avatar

    Join Date
    Aug 2004
    Posts
    15,806

    Ignore Hussar
    Quote Originally Posted by Blood & Bones View Post
    I think the answer is to have players rank their choices rather than pick one. This way, if nobody agrees with their number one choice, but everybody agrees on their number two choice, they get something they can live with rather than simply the choice that has the most votes (which most likely will not have a plurality vote).

    Does that make sense? I feel that I might be explaining it too obscurely.
    Yes, that makes perfect sense. Ok, like I said, I was just musing. It makes sense to springboard from the simple poll into more complicated ones.
    The rules don't give the DM their authority. The consent of the players does. - Mallus

  • #9
    Quote Originally Posted by Quickleaf View Post
    I've read the article where Mearls discusses the "Gnome Effect" (Dungeons & Dragons Roleplaying Game Official Home Page - Article (What's With the Polls?)) and I'm not sure I grasp what he's saying. Either it's painfully obvious (change the game and not only do individual players react but also gaming groups) or it's going over my head.
    The point of that article is that the polls on the WotC articles are not a form of market research. Due to a variety of reasons, including the "gnome effect", these polls are completely useless as a means of gathering information. They have a whole department (which interfaces with third party market research companies) to do that sort of stuff, and they're very good at what they do.

  • #10
    Registered User
    Cutpurse (Lvl 5)



    Join Date
    Dec 2006
    Location
    North Germany
    Posts
    709

    Ignore 1of3
    Quote Originally Posted by Hussar View Post
    So, therein lies the problem. If you are designing a game, what do you do??
    Set my polling plug-in for approval voting for once.

  • + Log in or register to post

    Similar Threads

    1. Replies: 0
      Last Post: Friday, 18th June, 2010, 02:21 PM
    2. EN Worlders Open Playtesting?
      By Mark in forum General RPG Discussion
      Replies: 6
      Last Post: Monday, 10th August, 2009, 02:52 AM
    3. [Open Design] Open Design Project #9 Seeking Patrons!
      By terraleon in forum General RPG Discussion
      Replies: 14
      Last Post: Wednesday, 28th January, 2009, 04:44 PM
    4. Open Design / Wolfgang Baur to do a 4E Open Design Adventure
      By Admiral Caine in forum Older D&D Editions and OSR Gaming
      Replies: 81
      Last Post: Tuesday, 24th June, 2008, 04:14 AM

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •