Are forums representative of users?

[Mark CMG]

Actually, it *is* a caution that negates the collective opinions from being accurately representational. That's the science of statistics. The self-selected group will generally be quantitatively and qualitatively different from the general population.

To say that a group represents the whole is more than just "all opinions that appear in the whole appear in the group". To be properly representational, the opinions of the whole appear in the group with the same frequency, with the same vehemence, and the group should not have opinions that don't appear in the general population.

If you poll a self-selected group, and you poll the general population, the poll results will usually be different. That means the one does not represent the other.

You can even often poll a self-selected group on things that seem entirely unrelated to the selection (for example, asking self-described "fiber artists" about politics) and still not have it match the general population. The biases can run deep and strange.

Actually, it *isn't* a caution that negates the collective opinions from being accurately representational when the sample size is large enough. But you've chosen to only quote part of what I said. Since you ignored a portion of my post where I discussed the sample size being large enough, you've managed to only be correct in regard to what you have posted, not in regard to what I posted in whole. The larger the sample size, the less likely self-selection is a factor in whether the data is qualitatively representational. Furthermore, as to whether it is quantitatively accurate, the larger the sample size, the less likely it is inaccurate as well though any individual opinion is still more difficult to assert as being representational of the same percentage of the whole. In any event, one has to determine if the sample size is large enough to feel the data can be accurate no matter whether you are dealing with a self-selected or a randomly selected sampling of the whole.

The existence of bias can not be predicted as more or less in evidence in self-selected groups then in randomly selected groups. And in point of fact, there is no way to eliminate bias in any type of group. You also cannot say definitively that the biases of any group, self-selected or otherwise, aren't possibly an accurate representation of the biases of the whole. No matter the sample size or the manner of selection, it is paramount to collect large amounts of data and use polling questions that are fashioned to lessen the impact of biases. To offhandedly toss out self-selection in a manner that overstates its importance whenever a discussion of how representional message board opinions can be is a narrow, incomplete view of the science of statistics.

 

[Kzach]

Actually, it *isn't* a caution that negates the collective opinions from being accurately representational when the sample size is large enough.

Sample size is irrelevant when talking about selection bias. The likelihood of bias is 100% no matter what the sample size because the selection is... and here's the kicker... BIASED! The percentage of bias remains the same no matter how large or small the sample size.

For sample size to have any relevance, there can't be any selection bias at all and since you can essentially never eliminate bias, statistics from surveys are basically irrelevant.

They sure do make purrdy graphs though.

 

[FireLance]

So to me, the statement: "...the changes with the most rage had little effect on business performance measures." - just doesn't add up...

The conclusion I draw from this is: people complain, but they still keep buying. It doesn't matter what they say; it doesn't correspond to what they actually do.

 

[IronWolf]

I don't think forum users are representative of the gaming population. I know in our group two of us hang out on EN World, but the others three do not. We game every week, so it we do have a reliable gaming group.

With that said, I do think publishers should hang out on forums (even as lurkers) to get a feel for what the forumites are saying and commenting on. While I wouldn't expect product changing decisions to be based solely on message board post praises and complaints I think they are a source of what at least one subset of your players think. It just needs to be taken with the grain that it is a very vocal subset of players.

 

[Yesway Jose]

There is a difference between being vocal about an opinion vs sharing that same opinion but not being vocal.

That is, if the 7 players at a gaming group think that "bloodied" is a misnomer (just to grab one example), and 1 guy gets incensed about it and complains on Enworld, then it's not unrepresentative of the group. If 4 of the other players don't care and 3 of the other players do care, the 1 forum user is still representative of 50% of that gaming group.

I understand there's a self-selection bias for vehemence, but not necessarily for the quality of the opinion.

You need a statistically significant results to draw a scientific conclusion, but not all conclusions have the luxury of being scientific.

 

[Dedekind]

Okay, some of this doesn't compute for me. Maybe it's just that the provided data seems incomplete. Based on just that data, one of two assumptions can be made and two conclusions drawn:

1. Either the 2% that post spend 40 times more than all other users/customers (as the 2% are the only one's that post, making all other customers those that don't post)
2. Or the 2% that post spend 40 times more than the other user/customers that read but don't post (18% of total users/customers).

Either way, if the 2% were spending 40 times more than all customers - that's a significant percentage of overall sales (the majority of revenue) and would be noticed if anything impacted the purchasing trends of that group...

Or, even if the 2% were only spending 40 times more than the 18% of readers-but-not-posters, that would still be a significant majority of revenue unless somehow the 80% of users/customers that don't read or post somehow spent more on average (per person) than the 20% of users/customers that read forums (whether they post or not) - and I'd sincerely doubt that.

So to me, the statement: "...the changes with the most rage had little effect on business performance measures." - just doesn't add up...
...

Sorry if you were confused. The presentation is very clear about it, though my memory may be a little unclear.

1) The total population average $.50 per month.
2) 75% of the population spent some money (I think that is right).
3) 20% of the population read the forums.
4) 2% of the population posted on the forums.
5) The 2% of the posting population averaged $20 per month.

The posters spent 40 times more than the entire population. That is slightly misleading, since the population average includes the posters and also people who don't spend anything at all. But, without applying maths, lets assume that the disparity between posters and the average player is only twenty times more. Seems economically important to me...

However, as to your point about the population representing a large sales, they go on to show that the posters continue to purchase even at the same point at which they are raging. When they introduced premium weapons (that were superior), everybody was mad... but they bought them.

 

[El Mahdi]

Sorry if you were confused. The presentation is very clear about it, though my memory may be a little unclear.

1) The total population average $.50 per month.
2) 75% of the population spent some money (I think that is right).
3) 20% of the population read the forums.
4) 2% of the population posted on the forums.
5) The 2% of the posting population averaged $20 per month.

So, the posters spent 40 times more than the entire population. That is slightly misleading, since the population average includes the posters and also people who don't spend anything at all. But, without applying maths, lets assume that the disparity between posters and the average player is only twenty times more. Seems economically important to me...

Cool. Thanks for clearing that up, especially since I didn't want to listen to the whole thing. And I agree, seems economically important to me too, unless FireLance's scenario is correct.

Mathematically, what they're saying is that the the 2% (posters) are contributing 80% of total revenue (with the other 98% of customers contributing 20% of total revenue).

Unless the sitution is as @FireLance said: that even when the 2% is expressing strong dissaproval (rage) on the forums, the 2% doesn't significantly alter their buying habits (the amount of revenue from them doesn't significantly change)...then I don't see how if a group that contributes 80% of total revenue suddenly starts spending less money, it doesn't cause a significant impact???

To make the statements they did, FireLance's scenario must be true (unless they aren't telling the whole truth). If it's true, then ignoring forum rage costs them nothing. Or maybe it's just a red herring to throw off the forum rage... (could they be that smart...)

I'd think though that some of this does cross over to RPG's, but not all. In gaming groups, the DM is usually the trend setter of the group (to varying degrees) - a factor that doesn't have an analogue with EA's games. So forum rage of a self-selected group that's mostly DM's, has at least some impact/influence on the non-forum represented customers. How much that impact is (in monetary terms) is anyone's guess.

So even though there are similiarities, it's still apples and oranges to an extent.

 

[UngainlyTitan]

Cool. Thanks for clearing that up, especially since I didn't want to listen to the whole thing. And I agree, seems economically important to me too, unless FireLance's scenario is correct.

Mathematically, what they're saying is that the the 2% (posters) are contributing 80% of total revenue (with the other 98% of customers contributing 20% of total revenue).

Unless the sitution is as @FireLance said: that even when the 2% is expressing strong dissaproval (rage) on the forums, the 2% doesn't significantly alter their buying habits (the amount of revenue from them doesn't significantly change)...then I don't see how if a group that contributes 80% of total revenue suddenly starts spending less money, it doesn't cause a significant impact???

To make the statements they did, FireLance's scenario must be true (unless they aren't telling the whole truth). If it's true, then ignoring forum rage costs them nothing. Or maybe it's just a red herring to throw off the forum rage... (could they be that smart...)

I'd think though that some of this does cross over to RPG's, but not all. In gaming groups, the DM is usually the trend setter of the group (to varying degrees) - a factor that doesn't have an analogue with EA's games. So forum rage of a self-selected group that's mostly DM's, has at least some impact/influence on the non-forum represented customers. How much that impact is (in monetary terms) is anyone's guess.

So even though there are similiarities, it's still apples and oranges to an extent.

I though it was 2% of the forum population that active posters and that their spending was unaffected by the changes.

I went back an listened to the forum user section the $22 is total lifetime spend by the forum poster the rate they out spend the average user is 10 times the regular user, which make then constitute about 20% of revenue.

 

[CuRoi]

When they introduced premium weapons (that were superior), everybody was mad... but they bought them.

This explains why I can't play MMORPGs. I vote with my feet, not my mouth so when something happens that drives a price beyond what I expect to pay, I'll gladly ditch it. In the end, the game producers end up catering to a very vocal minority with incredibly loose purse strings (or with full blown addicitions IMO) that apparently want to play a game (both in a rules and economic sense) that I have no patience for. Completely explains what has happened to just about every MMORPG I have tried to play.

Fascinating discussion guys!

 

[Aberzanzorax]

Interesting.

I'll say this anecdote. I'm the only forumgoer in my group and I probably spend more on that same order (I buy 2-6 rpg books per month, they each generally buy 1 book per year).