Menu
News
All News
Dungeons & Dragons
Level Up: Advanced 5th Edition
Pathfinder
Starfinder
Warhammer
2d20 System
Year Zero Engine
Industry News
Reviews
Dragon Reflections
Columns
Weekly Digests
Weekly News Digest
Freebies, Sales & Bundles
RPG Print News
RPG Crowdfunding News
Game Content
ENterplanetary DimENsions
Mythological Figures
Opinion
Worlds of Design
Peregrine's Next
RPG Evolution
Other Columns
From the Freelancing Frontline
Monster ENcyclopedia
WotC/TSR Alumni Look Back
4 Hours w/RSD (Ryan Dancey)
The Road to 3E (Jonathan Tweet)
Greenwood's Realms (Ed Greenwood)
Drawmij's TSR (Jim Ward)
Community
Forums & Topics
Forum List
Latest Posts
Forum list
*Dungeons & Dragons
Level Up: Advanced 5th Edition
D&D Older Editions
*TTRPGs General
*Pathfinder & Starfinder
EN Publishing
*Geek Talk & Media
Search forums
Chat/Discord
Resources
Wiki
Pages
Latest activity
Media
New media
New comments
Search media
Downloads
Latest reviews
Search resources
EN Publishing
Store
EN5ider
Adventures in ZEITGEIST
Awfully Cheerful Engine
What's OLD is NEW
Judge Dredd & The Worlds Of 2000AD
War of the Burning Sky
Level Up: Advanced 5E
Events & Releases
Upcoming Events
Private Events
Featured Events
Socials!
Twitch
YouTube
Facebook (EN Publishing)
Facebook (EN World)
Twitter
Instagram
TikTok
Podcast
Features
Top 5 RPGs Compiled Charts 2004-Present
Adventure Game Industry Market Research Summary (RPGs) V1.0
Ryan Dancey: Acquiring TSR
Q&A With Gary Gygax
D&D Rules FAQs
TSR, WotC, & Paizo: A Comparative History
D&D Pronunciation Guide
Million Dollar TTRPG Kickstarters
Tabletop RPG Podcast Hall of Fame
Eric Noah's Unofficial D&D 3rd Edition News
D&D in the Mainstream
D&D & RPG History
About Morrus
Log in
Register
What's new
Search
Search
Search titles only
By:
Forums & Topics
Forum List
Latest Posts
Forum list
*Dungeons & Dragons
Level Up: Advanced 5th Edition
D&D Older Editions
*TTRPGs General
*Pathfinder & Starfinder
EN Publishing
*Geek Talk & Media
Search forums
Chat/Discord
Menu
Log in
Register
Install the app
Install
Community
General Tabletop Discussion
*TTRPGs General
Interesting Ryan Dancey comment on "lite" RPGs
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Reply to thread
Message
<blockquote data-quote="Steve Conan Trustrum" data-source="post: 2496354" data-attributes="member: 1620"><p>You have to keep in mind that how this information is gathered is sample polling rather than complete polling. This means that they aren't calling every gamer and getting the information (and boy wouldn't THAT be fun!) Instead, they call around to gamers and/or shops and take information from a small sample of the demographics they are surveying. They then take that data and say "well, we got 2000 replies and we figure there are probably 2,000,000 gamers in the US, so that means every reply will account for the trends of 1000 people' (that's a simplification of the process, but it gives you an idea.) Obviously, the larger the sample the more accurate your results--surveying two people and saying they each represent 1,000,000 people in the relevant market, for example, is not going to yield accurate data.</p><p></p><p>This is, of course, assuming all demographics are equal. Polling urban centers alone willl yield inaccurate results, as will only polling one part of the country. Budget also skews results because you may not spend the money on getting the sample you need, so you make do with what you can get. There are any number of things that can cause similar problems and,unfortunately, the client's employees in charge of research almost always don't understand those issues.</p><p></p><p>For example, I once fielded a project for one of North America's largest automotive manufacturers. I was managing the call centre at the time and was also in charge of how the call sample was acquired and fielded. The people heading up the project for the client simply didn't understand why we could easily fill their quota of mini-vans and SUVs when calling suburban areas but were having a hell of a time finding truck owners in those same areas. Likewise, when caliing numbers in the farmlands they didn't know why we could easily find people who owned trucks but were having a hard time finding people intending to buy compacts. I spent many hours trying to explain to these people--people who, working for the manufacturer, let alone their research department, should know this stuff--why fewer people own heavy-duty pickup trucks in the suburbs and why compacts/minis aren't common in the farmlands. These people were then responsible for taking the data we gathered and analysing it to prepare the report for their bosses. As you can imagine, these people likely drew a lot of false conclusions from the raw data and presented them as fact. This is pretty common in big companies because, as Ryan did, the people doing the analysis aren't experienced or trained enough with regards to research to recognize problems with their methodology.</p><p></p><p>EDIT: It's also worth nothing that you cannot make <strong>ANY</strong> accurate, statistical conclusions from qualitative research, such as Ryan conducted with the stop wathes. Why? Because you can't draw statistics from non-standardized surveying. His experiment was not standardized because each group consisted of different people being exposed to changing stimuli that wasn't the same for the other groups (in other words, the things the other participants said and did wasn't exactly the same for all the groups.) You can only draw accurate statistical conclusions from quantitative data where the questions are all standardized and not open-ended. (In other words, people are presenting you with facts and not opinions.) The only stats you can draw from qualitative work is that directly related to those groups, such as "4 out of 5 of the people in our 8 pm group didn't like X" but that data does not then transfer over to the general populace. It can only be used by the observers to make guesstimations.</p></blockquote><p></p>
[QUOTE="Steve Conan Trustrum, post: 2496354, member: 1620"] You have to keep in mind that how this information is gathered is sample polling rather than complete polling. This means that they aren't calling every gamer and getting the information (and boy wouldn't THAT be fun!) Instead, they call around to gamers and/or shops and take information from a small sample of the demographics they are surveying. They then take that data and say "well, we got 2000 replies and we figure there are probably 2,000,000 gamers in the US, so that means every reply will account for the trends of 1000 people' (that's a simplification of the process, but it gives you an idea.) Obviously, the larger the sample the more accurate your results--surveying two people and saying they each represent 1,000,000 people in the relevant market, for example, is not going to yield accurate data. This is, of course, assuming all demographics are equal. Polling urban centers alone willl yield inaccurate results, as will only polling one part of the country. Budget also skews results because you may not spend the money on getting the sample you need, so you make do with what you can get. There are any number of things that can cause similar problems and,unfortunately, the client's employees in charge of research almost always don't understand those issues. For example, I once fielded a project for one of North America's largest automotive manufacturers. I was managing the call centre at the time and was also in charge of how the call sample was acquired and fielded. The people heading up the project for the client simply didn't understand why we could easily fill their quota of mini-vans and SUVs when calling suburban areas but were having a hell of a time finding truck owners in those same areas. Likewise, when caliing numbers in the farmlands they didn't know why we could easily find people who owned trucks but were having a hard time finding people intending to buy compacts. I spent many hours trying to explain to these people--people who, working for the manufacturer, let alone their research department, should know this stuff--why fewer people own heavy-duty pickup trucks in the suburbs and why compacts/minis aren't common in the farmlands. These people were then responsible for taking the data we gathered and analysing it to prepare the report for their bosses. As you can imagine, these people likely drew a lot of false conclusions from the raw data and presented them as fact. This is pretty common in big companies because, as Ryan did, the people doing the analysis aren't experienced or trained enough with regards to research to recognize problems with their methodology. EDIT: It's also worth nothing that you cannot make [b]ANY[/b] accurate, statistical conclusions from qualitative research, such as Ryan conducted with the stop wathes. Why? Because you can't draw statistics from non-standardized surveying. His experiment was not standardized because each group consisted of different people being exposed to changing stimuli that wasn't the same for the other groups (in other words, the things the other participants said and did wasn't exactly the same for all the groups.) You can only draw accurate statistical conclusions from quantitative data where the questions are all standardized and not open-ended. (In other words, people are presenting you with facts and not opinions.) The only stats you can draw from qualitative work is that directly related to those groups, such as "4 out of 5 of the people in our 8 pm group didn't like X" but that data does not then transfer over to the general populace. It can only be used by the observers to make guesstimations. [/QUOTE]
Insert quotes…
Verification
Post reply
Community
General Tabletop Discussion
*TTRPGs General
Interesting Ryan Dancey comment on "lite" RPGs
Top