Broken Base Lookback: Editions

[BryonD]

However, the broken base is more of an economic thing.

I think this is the key point.


But beyond that:
There have always been divisions. But it is the shape of the divisions that should be considered, not just their presence.

At the height of 3E, anyone who claimed there was not a split in the D&D community simply had their head in the sand. But it really wasn't a topic of conversation. The proportions of the split that are important. I don't think any of the anti-3E holdouts were any less emotional, nor any less valid in their opinions, than anti-4E holdouts now. The qualities of the disagreement were ultimately highly similar. But I doubt anyone was claiming that those differences constituted a significant split in terms of market share.

Trying to please everyone is a plan for failure.
Trying to cast as large a net as possible, on the other hand, is a good idea.
Confusing one for the other is a mistake.

It is tempting to point out that the quotes in the OP are all from the roll out period of 3E. When 5E comes out X years from now the same type of doom-saying will come. But if X+3 years later the doom saying is bearing out for a lot of the market, that will be bad. If, on the other hand, X+3 years later it is just a few voices in a market that is clearly dominated by 5E, then the unavoidable fact that some people are displeased will be both insignificant and generally not even worthy of conversation.

 

[Umbran]

Do you argue that I should stick with my assumption?

No. But I'm not arguing that you should always stick with assumptions.

I'm arguing that the generalization that "any data is better than no data" is demonstrably false. That's all. Any further steps you take with it are your own, not mine.

In the scenario above, as a scientist I'd not take the Bradbury number, or the single reading. Unless there were some emergency which required me to use an estimate of the temperature *NOW*, I'd admit, quite flatly, that I didn't have enough information to make an assertion with any confidence.

That's the trick. You don't actually have to take up one side or the other. "I don't know," is allowed.

 

[Desdichado]

I gotta say, "let's throw out the data because it's not good enough, and then since we have no data my assumptions must be true" really, really annoys me. Any data is better then pure assumptions. I've spent a lot of time on a couple occasions collecting data, and it really doesn't encourage me to do so for people who would rather believe in their assumptions than look at the data.

Y'know, I can't really be in the loop about every bit of fan data gathered out there. I think the ICv2 article was interesting.

But you make a huge mistake when you say that any data is better than just assumptions. Data, if it's incomplete, if there are problems with how its gathered or interpreted, can actually be worse than assumptions. I've seen plenty of data that, taken on its own, would seem to encourage the exact opposite conclusion from what whole giant stacks of other data suggest.

Data's not a magic word. And the conclusions you draw (in general, not talking about you specifically) are only as good as your data. If your data has sampling errors, endogeneity problems, bias, or other problems (and I think that this data certainly does,) then we might be actually better off not having it. You have to be very careful about interpreting data unless it's really incontrovertible data. You give me actual sales by month for many months running of 3e, 4e and Pathfinder, I won't have any argument. That's data you can take to the bank. LibraryThing records is not. There's no way to establish a causal link between library holdings and sales to the public. Frankly, there's also no way to establish a link between sales and what's actually being played.

And even the ICv2 article makes a point of pointing out the weaknesses of the data it presents. I don't know that this "data" changes my mind, but I do admit that it certainly is suggestive enough that I can consider that the prevailing wisdom position might not be true anymore. I'm not convinced that that's the case, but I am convinced that it could be. That's all.

 

[prosfilaes]

I'm arguing that the generalization that "any data is better than no data" is demonstrably false.

I'm arguing that you haven't demonstrated that.

In the scenario above, as a scientist I'd not take the Bradbury number, or the single reading. Unless there were some emergency which required me to use an estimate of the temperature *NOW*, I'd admit, quite flatly, that I didn't have enough information to make an assertion with any confidence.

I accept it as a philosophical position, but I do not see it as innately superior. Certainly if you're working on Venera 5, claiming that the single data points that Venera 4 gave you don't tell you anything about how to make Venera 5 survive Venus's atmosphere isn't going to let you keep your job.

That's the trick. You don't actually have to take up one side or the other. "I don't know," is allowed.

Fine. But why are you telling me that? I haven't taken a position in this thread. I was replying to someone who had taken a position, pointing out that he didn't have evidence for that position. Why didn't you tell him that "I don't know" is allowed?

Frankly, I see you as having taken a position. In my experience, I've seen you take an aggressive stance against data for Pathfinder in other threads. In that thread as well as this one, you've supported people who simply assume that D&D 4 is the clear leader versus people who have data for Pathfinder.

Data, if it's incomplete, if there are problems with how its gathered or interpreted, can actually be worse than assumptions.

You can produce the correct population for the United States by putting up 10 dart boards with the numbers 0-9 on them, and throwing darts. It's just massively improbable. The question here is not can data be wrong, but whether it is more reliable than assumptions.

Again, part of my assumptions about relative success of Pathfinder versus D&D 4 come from seeing that there were more Pathfinder Society games then LFR games at Neoncon. That's not good data; so why does it make for good assumptions?

I've seen plenty of data that, taken on its own, would seem to encourage the exact opposite conclusion from what whole giant stacks of other data suggest.

Sure. I've seen plenty of assumptions that were dead wrong, too. The question is not can either of them be wrong, but what's more reliable.

And the conclusions you draw (in general, not talking about you specifically) are only as good as your data.

Thanks for making my case for me. Assumptions are built off lousy data, so the conclusions you draw from your assumptions are lousy.

You have to be very careful about interpreting data unless it's really incontrovertible data. You give me actual sales by month for many months running of 3e, 4e and Pathfinder, I won't have any argument. That's data you can take to the bank. LibraryThing records is not.

I'm not taking data to the bank. I'm collecting a wide variety of data to give me as good an image of my world as I can get. It's not great data, but I think there's value in looking at the data we can get and trying to dig out what we can know instead of standing back and saying I don't know. Umbran disagrees, and I acknowledge his epistemological position, even if I don't hold it. If you want to say "I don't know", go for it. But you're saying:

it certainly is suggestive enough that I can consider that the prevailing wisdom position might not be true anymore.

Prevailing wisdom is not data you can take to the bank. Prevailing wisdom is crap. Heck, how do you even know that your position is prevailing wisdom? How can you lecture me about "incontrovertible data" and hold to prevailing wisdom?

 

[Hussar]

In a recent interview on FtB Fear the Boot Blog Archive Interview Episode 20 – Erik Mona Erik Mona commented on the ICv2 stats. Paraphrasing it, it comes out to, "This is the best of the crappy information that we have". And that's about it.

It's significant in that it shows that in one area, Pathfinder is doing very, very well. Great for them. But, it's hardly much more than a corner piece and a little bit of the wobbly bits on the right hand side of the puzzle. After all, how did the WOTC Basic set do at Target? That's not tracked at all. We have no idea. Yet, that's a major release through a huge distrubutor for WotC. Never minding the online elements as well.

I will agree with BryonD on one thing though. The topic of the split certainly gets a lot more air time now than it did, say, five years ago. Although, to be honest, 5 years ago, it would be framed as, "Well, I used to play 3e, but now I play ____ by some 3rd party publisher which is far more creative than anything coming out of WOTC."

 

[Desdichado]

Prevailing wisdom is not data you can take to the bank. Prevailing wisdom is crap. Heck, how do you even know that your position is prevailing wisdom? How can you lecture me about "incontrovertible data" and hold to prevailing wisdom?

Because the prevailing wisdom is drawn from the last time real data was released publicly, c. 1999 or 2000. Therefore, it's automatically the null hypothesis.

You haven't demonstrated via your data that the null hypothesis is wrong. You've certainly suggested it, but you haven't proven it.

Really, this whole "argument" is quite silly. Are you familiar with statistics and how they work? You can't just say "I have some numbers, therefore I'm right" which seems to be the gist of your position.

 

[Umbran]

I'm arguing that you haven't demonstrated that.

In terms of logic, I have not only demonstrated it, but outright proven it.

"Any data is better than..." is an absolute statement. All I have to do is present a single counter-example to disprove it. I have done so. But, let me make it even more clear:

You want to know the average temperature in August on the National Mall in Washington DC - you assume it will be hot, but you want to know more accurately and precisely than that.. You take two measurements: on the first and last day, and average them. Your thermometer is broken, and your data says it was -20 degrees Celsius.

There you go - a single counter-example, in which data is not better than assumption. QED.

Now, you can argue that your particular data is better than a particular assumption - and that's fine. I encourage that, because that's sound intellectual process. But I don't accept "any data..." as a valid argument. Nobody should, because it simply isn't true. Like life, a datum unexamined is not worth having.

I accept it as a philosophical position, but I do not see it as innately superior. Certainly if you're working on Venera 5, claiming that the single data points that Venera 4 gave you don't tell you anything about how to make Venera 5 survive Venus's atmosphere isn't going to let you keep your job.

I can't tell if you are missing my point, or misrepresenting it.

I will say, again, because it had no apparent impact earlier - I do not argue that assumption is always better than data. Nor do I argue that data tells you nothing at all. I argue that you cannot assume that because it is "data", that it is better than assumption.

It isn't as if they didn't examine data from Venera 4 - they were quite aware that, under those conditions instruments could (in fact, were certain to eventually) fail, or give anomalous readings. Given that examination, the data from Venera 4 is not just "any data" - it is data of known pedigree and well-stated limitations. And even then, no responsible researcher takes the data from Venera 4, and extrapolates long-term averages from it alone.

Frankly, I see you as having taken a position. In my experience, I've seen you take an aggressive stance against data for Pathfinder in other threads.

Yes. But note that I take an aggressive stance against the data, not against Pathfinder. There's a big difference. I think Pathfinder is an okay game. I think 4e is an okay game. I own books for both, and I'll play either, and have a good time.

You were the one who said, "any data is better..." That's false, so I took issue with the point. If the other guy had said it, I'd have taken it up with him. I take issue with folks who act as if ICV2 and Amazon sales "data", because it is shoddy data, not because I care whether WOtC or Paizo is ahead. Heck, recently on these boards, I took issue with representation of data that supported a position I happen to believe is true (that Americans should get out of their chairs and exercise more).

It isn't about Paizo and WotC, for me. It is about critical thinking and how to use data properly, in general.

The question here is not can data be wrong, but whether it is more reliable than assumptions.

Yes. And that question has to be asked. It should not be taken for granted that the data in hand is better, or you are just as guilty of making assumptions as the other guy.

 

[prosfilaes]

Really, this whole "argument" is quite silly. Are you familiar with statistics and how they work? You can't just say "I have some numbers, therefore I'm right" which seems to be the gist of your position.

I am familiar with statistics. You've missed my position. My position is that if you want to say "I don't know", that's fine, but if you want to take a position, it should be based on data, not assumption.

Because the prevailing wisdom is drawn from the last time real data was released publicly, c. 1999 or 2000. Therefore, it's automatically the null hypothesis.

Yay! I'm doing the happy dance! Finally we're out of the assumption area and we're talking about data!

Taking the real data (not the prevailing wisdom) as the null hypothesis is entirely reasonable. But I submit that the data you're relying on tells you very little.

The data: Adventure Game Industry Market Research Summary (RPGs) V1.0 -- Wizards of the Coast

In particular:

Getting back to the people still playing the games, when asked what games TRPG players play monthly, the answers (multiple choices allowed) were: D&D: 66% Vampire: The Masquerade: 25% ...

With Werewolf: The Apocalypse at 15% and Alternity at 4%. We have WotC games at 66-70% of the players and WW games at 25-40%. Hardly 800 lb gorilla material.

Notice also that it says D&D, not AD&D. This survey was done in 1999, before D&D 3 came out, so they're obviously lumping in all editions of D&D and AD&D. So taking the D&D numbers as your D&D 4 numbers is obviously incorrect. How many people there were playing an in-print version of D&D? It doesn't say.

While D&D players 25-35 spent $1600 a year on their RPGs ($500 less than RPers as a whole), they also tell us that non-miniature players (all ages) averaged $139, with miniature players spending $4,413, so I'd say a lot of that is miniatures. (And possibly used material and books for non-D&D systems.) Extrapolating forward, even if they were buying new D&D books, would they continue to buy them from WotC once WotC gave up their monopoly? Would they change to a new system if their old system was still supported by D20 companies?

The most literal interpretation of this survey is that nobody plays D&D 3, 4 or Pathfinder, since none of the people they surveyed did. When extending this, should we conclude that Pathfinder would fall under D&D or not? Yes or no, I think we're assuming what we're trying to prove. When a patent or copyright expires or is given up, generics and other brands come into the market, so I see no reason to assume the OGL & SRD didn't change things.

Basically, we have a 12-year-old figure for the number of people who played D&D, all editions. It is completely wrong to extrapolate that to people who play just the current edition of D&D; that's not what the original data was measuring. The changes that the OGL & SRD wrought on the market don't incline me to assume the D&D numbers then don't apply to the sum total of all D&D derived games now.

 

[prosfilaes]

You want to know the average temperature in August on the National Mall in Washington DC - you assume it will be hot, but you want to know more accurately and precisely than that.. You take two measurements: on the first and last day, and average them. Your thermometer is broken, and your data says it was -20 degrees Celsius.

There you go - a single counter-example, in which data is not better than assumption. QED.

As a purely logical argument, it works. As a statement about reality, it's crap. I don't assume that it will be hot. I know from data that it will be hot. Why? Because Washington DC is approximately between the South and the North of the United States; as it's in the Northern Hemisphere, it generally gets colder as you go north. Cities in proximity to the ocean, especially the Gulf Stream, are going to have more moderate temperatures. As a general rule of thumb, a coastal city south of Boston is going to be warmer at the same time of year then Boston is, and I've been in Boston in August, and it's been 20-30 Celsius. I've seen temperature maps of the US before, and never opened the newspaper and seen DC or anything on the eastern seaboard at subzero temperatures in summer. And I've heard descriptions of DC in the summer as hot and muggy. Once I add all this data up and the two readings of the thermometer, it's obvious that the thermometer readings are completely anomalous.

I don't ask that anyone swallow the Amazon numbers or any other numbers as truth. I do ask that you choose not to say "I don't know" and take a position for it you have data to support it. If it goes against the numbers presented, you should be willing to justify why you believe the data for your position to be better than the data against your position.

I took issue with representation of data that supported a position I happen to believe is true (that Americans should get out of their chairs and exercise more).

Why was it a representation of data, instead of a representation of assumption? Not only that, you weren't nearly as critical of the data you gave as you are of the Pathfinder data; 1.4 calories / hour is for a 150 lb person quietly watching TV. There, you were willing to toss out data in support of your claim; here you criticize anyone tossing out data.

 

[Desdichado]

Basically, we have a 12-year-old figure for the number of people who played D&D, all editions. It is completely wrong to extrapolate that to people who play just the current edition of D&D; that's not what the original data was measuring. The changes that the OGL & SRD wrought on the market don't incline me to assume the D&D numbers then don't apply to the sum total of all D&D derived games now.

Oh, I realize that. I'm just not sure that any data that's been made available is sufficient to say that the 12 year old picture has changed substantially. There's a lot of reasons it could have changed, but it seems even now that the impact of those changes can only really be speculated on, not spoken to authoritatively.

I agree that the Amazon and ICv2 data suggests that it may have changed, and may suggest that Pathfinder is selling in the same range as 4e. Maybe. But the data--again--seems to have a number of issues. And, as I mentioned earlier, even sales data don't tell you what people are playing.

So, I think your data is suggestive. But I reserve judgement. The null hypothesis, i.e., that D&D 4e is the 800 lbs gorilla of the market just as its predecessor was 12 years ago, still remains a compelling story. Especially considering the non-data yet observational input that I still see D&D on the shelves in stores much more than I see anything else. By a huge factor.

Including Pathfinder.