Are forums representative of users?

[Umbran]

Actually, it *isn't* a caution that negates the collective opinions from being accurately representational when the sample size is large enough.

The behavior of a sample is expected to reflect the population as the sample size grows only if the sample is taken randomly from the population*.

By definition, a self-selected group is not taken randomly from the population. They are selected from the population by their choices.

The posters of EN World (or any other particular messageboard, or even all messageboards together) are not chosen at random from the gamer population, so they are not a representative statistical sample, period. It really is that simple.

There are ways to correct results when a representative sample is not available, but they are mathematically complex, and generally require you know in detail how your sample differs from the general population.

*Edit: I should add - this is where the term "selection bias" comes from - it is the bias in results that comes from non-random sample selection. "Self-selection bias" is just what you get when the sample selects itself out of the randomness, as opposed to when the researcher does it.

 

[Dedekind]

This explains why I can't play MMORPGs. I vote with my feet, not my mouth so when something happens that drives a price beyond what I expect to pay, I'll gladly ditch it. In the end, the game producers end up catering to a very vocal minority with incredibly loose purse strings (or with full blown addicitions IMO) that apparently want to play a game (both in a rules and economic sense) that I have no patience for. Completely explains what has happened to just about every MMORPG I have tried to play.

Fascinating discussion guys!

I totally agree that I wouldn't fit into this scenario at all. The guy offers an amazingly frank discussion about the player psychology. The developers were dead set against offering anything that gave you an advantage in game. Once they did some market research, they found that a significant group would be willing to pay for premium weapons (like 40+% of responders). When they introduced it, it had little effect on the number of players but increased revenues.

His interpretation was that people with premium stuff were usually either better at the game or posers. The free players didn't care to be beat by better players and they were motivated to beat the posers who just had more money. This is supposition, of course.

Some of it comes down to how premium is premium. On average, they guessed it was a 10% increase in power. One example was a sniper rifle that came with a 7 cartridge magazine instead of a 5 cartridge magazine. I wouldn't feel too bad about playing in that environment with the free weapons, but I could see how others would.

To match this more to TTGs, it is kinda like those rare Magic cards or the randomized miniature packs. If you were willing to buy enough base packs, you could build a nice deck/army/what-have-you. You know your chances aren't as good against somebody with the complete set, but Is their model that much different?

 

[Dedekind]

The behavior of a sample is expected to reflect the population as the sample size grows only if the sample is taken randomly from the population.

...

There are ways to correct results when a representative sample is not available, but they are mathematically complex, and generally require you know in detail how your sample differs from the general population.

Absolutely correct. The statistics we usually bring to bear on a problem are all based on dealing with sampling error--the error that comes from having a random sample of the full population. If it isn't a random sample, then the statistics are useless. Forums have people who, by definition, care enough to post and so probably more dedicated (and therefore more invested) than the population. Ergo, not a random sample and (most likely) not representative.

Given all that, I wouldn't say the forum opinions are not present in the general population. I would say the non-forum opinions are, on average, ceteris peribus, not as strongly held.

Give McCloskey and Zilliak a read, the first three chapters are very accessible:

[ame=http://www.amazon.com/Cult-Statistical-Significance-Economics-Cognition/dp/0472050079]Amazon.com: The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives (Economics, Cognition, and Society) (9780472050079): Stephen T. Ziliak, Deirdre N. McCloskey: Books[/ame]

 

[Dedekind]

I though it was 2% of the forum population that active posters and that their spending was unaffected by the changes.

I went back an listened to the forum user section the $22 is total lifetime spend by the forum poster the rate they out spend the average user is 10 times the regular user, which make then constitute about 20% of revenue.

Thanks for double-checking. I didn't have the time to look.

So, the people who were spending were also posting, and didn't change after the premium weapons went in.

The people who weren't spending also weren't posting, but did spend after the weapons went in.

Did I summarize that correctly?

 

[UngainlyTitan]

Thanks for double-checking. I didn't have the time to look.

So, the people who were spending were also posting, and didn't change after the premium weapons went in.

The people who weren't spending also weren't posting, but did spend after the weapons went in.

Did I summarize that correctly?

yeah, that about sums it up.

 

[Mark CMG]

A sample size of 100% of the whole, in essence the entire populace under consideration, removes entirely the effects of self-selection, but the bias even of an entire populace can never be removed whether the group is self-selected or randomly selected, even when the sample is 100% of the populace. Some people conflate their interpretation of data with what the science of statistics is meant to allow, in an effort to seemingly allow no interpretation of data at all that can be taken as meaningful. The science of statistics is meant to guide interpretation not remove interpretation entirely.

 

[Kzach]

A sample size of 100% of the whole, in essence the entire populace under consideration, removes entirely the effects of self-selection, but the bias even of an entire populace can never be removed whether the group is self-selected or randomly selected, even when the sample is 100% of the populace.

100% of WHAT whole? The whole world? The whole supermarket? The whole school? The whole forum member database?

100% of a sample that is biased towards one thing or not gives you a biased result.

Take shoppers, for instance. If I ran a survey in a Woolworths Supermarket, and asked 100% of shoppers who entered the supermarket for seven days to cover the whole week and for 24 hours out of every day, is that then a 100% of the whole?

Being 100% of the whole, does that then represent all shoppers, everywhere? Or does it only represent Woolworth's shoppers? Does it represent all families of those shoppers?

At what point do you determine where the bias ends?

In all the above examples, there is selection bias. Just selecting what 'whole' you're taking your sample from, gives it a bias.

 

[Starman]

This is where I feel WotC really lets itself down. I feel they pay FAR too much attention to the posts of forum users. In effect, I think it's the lazy-man's customer feedback. You see feedback and think a forum is a useful means of feeling the pulse of your customer base when in truth they're only a very small minority of vocal users who aren't even representative of the overall customer base.

Maybe that's true up to a point, but if forum posters spend more on average than non-forum posters, shouldn't the forum poster's views carry a bit more weight?

 

[UngainlyTitan]

Maybe that's true up to a point, but if forum posters spend more on average than non-forum posters, shouldn't the forum poster's views carry a bit more weight?

Only if;
a) what the forum posters say affect their spending habits, which in the case of the talk linked to in the OP they did not.

b) You have no way to get the non posters/low posters to spend more money.

 

[Umbran]

A sample size of 100% of the whole, in essence the entire populace under consideration, removes entirely the effects of self-selection, but the bias even of an entire populace can never be removed whether the group is self-selected or randomly selected, even when the sample is 100% of the populace.

I think you may be using a different definition of "bias" than I.

A statistical bias comes from systematic error in measurement and/or calculation. Bias is something that is introduced by how you take your data or crunch your numbers. It is not in the subject of your measurement.

Say I have a box of toothpicks, and I want to know their length. I could measure all of them, and take an average, and find it to be 2.5". I would not say the toothpicks have a bias toward being 2.5" length.

Colloquially, we may say that a person or group of people is "biased", but that's a bit different than what I was talking about above.

There are errors that can arise in statistics that are not biases, and some of those can arise even if your sample is the entire group, yes.