ColonelHardisson said:
It must be. No other game company ever tried such extensive research before. Given 3e's success, maybe more should.
1) "Extensive" research doesn't count for much if it isn't done properly. Research that borders on social experimentation is one of the most difficult types of research to formulate properly, with product assessment being one of the easiest (in other words, organizing research where people look at a product and say why they think it looks better in blue than in pink is a lot easier than trying to recreate a complex social situation and get reliable results.)
2) Most of WotC's research is what's known as quantitative, which is the most common and most easily controlled type of research. Putting people in a room and asking them to recreate a social situation while you time them is known as qualitative research and is the most difficult to pull off properly. The much-referred to survey that gathered information on rpg use versus other games (among other things) is quantitative and has nothing to do with the focus groups Ryan first mentioned.
3) Most of WotC's research (that they've made public, anyway) is about rpgs in general and is not about 3e. This shows in the game. I think d20 is great, but there's nothing revolutionary about it except for the OGL. That has absolutely nothing to do with the game mechanics inside. And this ties right into point 4 ...
4) Make no mistake that 3e's popularity is largely attributed to two things a) the biggest case of brand recognition existing in the industry and b) the OGL. The kind of research Ryan is talking about in relation to this thread--that of game mechanic and design--has nothing to do with either of those factors.
Never confuse research brand tailoring and product design. The two types of research MUST be conducted under different conditions, in seperate trials, in order to be considered reliable because they necessarily approach the same product from two distinctly different perspectives.
I do, however, agree with you that the industry would benefit a hell of a lot from some good market research. Market research is, however, prohibitively expensive to everyone in the industry except the top two or maybe even three companies in industry. Proper research will easily eat the budget of a nice sized book that is given a print run sized in the expectation of good sales.
For example, a single focus group room runs about $1000 (minimum) for just an afternoon or morning at a reputable facility. Want it for the whole day? Well, let's tack on an extra $500, bringing it to $1500 USD/day.
Now you need to pay for recruiting. Cold calling people and asking for role-players is going to be a tough recruit, but we'll still conservatively price it at $60 a recruit. We could tack on advertising budget to try and get some gamers to call in to see if they qualify, but we'll keep it simple.
To get a good idea of the market, let's cover the following age ranges: 14 to 17, 18 to 24, 25 to 34 and 35+. Again, to keep things simple, we'll not worry about education, income, etc. and go for a wide spread. We cannot, however, dismiss gender. Because most gamers are male we'll only make one group all females to stir things up a bit. The rest will be an all male group and a gender mixed group for each age range, bringing us to a total of 8 focus groups, each at the industry standard of 2 hours in length.
We'll also go for a typical group size, recruiting 12 people per group for 10 to show (meaning we over recruit so we have backups as there are typically no shows and we want to get as many full groups as possible.) 8 groups x 12 people (yes, you still have to pay the people who are sent home as overflow) means 96 people need to be recruited for the study.
Now we need to pay our people their incentive for coming to the focus groups. We'll use the industry standard of $50 a person for 2 hours of their time. 96 x $50 is $4800 in incentives. These 96 people also cost 96 x $60 to recruit, coming to $5760.
We could cram all 8 groups in a single day, but we'll go with what most research companies do for such a work load and spread it over 2 evening's (cramming it in can bias the results because the observers can become overloaded and bored, skewing what they perceive.) At
$1000 an evening, that now comes to $2000 for the room. The cost of the room typically includes the hostess, on-site rescreening of the recruiting data, taping, etc.
Now there are also going to be incidentals. Researchers rarely want to have their respondents come in pissed off, and they don't starve their observers, so we'll tack on a conservative estimate of an additional $500/day for food, drinks, snacks, etc. for a total of $1000.
Okay, let's look at the numbers here, keeping in mind I'm using typical costs and not accounting for problems, such as tough recruiting (which may account for a necessary increase in the incentive, paying more per recruit, etc.) I'm also being very generous with the amount of groups; considering that because this reserach doesn't happen regularly there should definately be more groups to get good data a good researcher would try for more than just the 8 groups. They'd also try and spread them out geographically to see if the data varies from different parts of the country (well, the world because rpgs are a global product.) But, as I said, we're being generous here.
The total cost for our generously devised, hypothetical market research is $13,560. I think you can see why most rpg companies don't run focus groups.
Now, a stint of quantitative research can be done much cheaper -- mailing out and/or cold calling surveys generally costs less per respondent, but because the individual results aren't as telling and suffer from greater limitations a lot more people have to be contacted for useful results. You'd also have a hard time using such a study for gathering anything other than hard, cold statistical data, so comparing rules heavy to rules lite in pretty much any terms other than "which do you prefer" or "how often do you play rules heavy as opposed to lite" is almost entirely out of the question.
