D&D General Todd Kenreck Let Go from WotC

[Ulorian - Agent of Chaos]

You are coming off as weirdly demanding that people have a debate for your benefit when Staffan said he wasn’t an expert on the topic.

Err... wut? @Belen makes a statement. @Staffan responds. I have almost no knowledge in this area, so asked these two to explain their positions, because it seems like an interesting topic. Belen responds with a super cogent, highly detailed response. Staffan responds with what amounts to 'I don't have a firm grasp on this topic either'. I agree with his sentiment, but it looks like he is arguing strictly based on 'feels', barring further explanation. Explain the 'weirdly demanding' aspect to me again?

I am jumping in as someone that was also rubbed the wrong way with the initial comment of “open access BS” which came off as dismissive of the concept as a whole but the follow up seems more reasonable in objecting to how access has been implemented than in the goal itself.

Cool! That's what I got from the response as well.

 

[The Firebird]

It is a mess. Governments fund research. The average person is not going to understand it. People should pay for the value added peer review, production, marketing, hosting and dissemination. That is work that was not funded. Let the un-peer-reviewed, raw papers be free with disclaimers on them that they have not been been peer reviewed.

I think it's reasonable to ask if this is actually adding any value. Peer review now functions more as a hoop to jump through and a way for reviewers to push their hobby horses than something that actually improves research. Typesetting and production are nice, but few people use physical journals and people can learn to typeset their own work--if not journal quality, not that far off. It isn't that difficult these days. Hosting also has alternatives like ArXiv. Personally I don't think much would be lost by scrapping peer review altogether.

The main issue now, imo, is that people see "peer review" as a mark of quality when it is not--it can be done by people who are not experts in the field and does not include attempts to replicate key findings or rerun data analyses. Then you get these predatory journals claiming "we peer reviewed X" as if that gets them anywhere.

"Public posting, public comment" would be a stronger paradigm. Anyone can publish anything they want, and anyone can comment on those findings. The authors of course have the chance to revise in response to those public comments.

You would lose some of the signaling benefits of journals -- i.e., the editors of publication X thought this met their standards. But there are other ways to achieve the same -- notes of 'I liked this paper' from the same senior authors who would be peer reviewing, for example.

 

[Belen]

I think it's reasonable to ask if this is actually adding any value. Peer review now functions more as a hoop to jump through and a way for reviewers to push their hobby horses than something that actually improves research. Typesetting and production are nice, but few people use physical journals and people can learn to typeset their own work--if not journal quality, not that far off. It isn't that difficult these days. Hosting also has alternatives like ArXiv. Personally I don't think much would be lost by scrapping peer review altogether.

The main issue now, imo, is that people see "peer review" as a mark of quality when it is not--it can be done by people who are not experts in the field and does not include attempts to replicate key findings or rerun data analyses. Then you get these predatory journals claiming "we peer reviewed X" as if that gets them anywhere.

"Public posting, public comment" would be a stronger paradigm. Anyone can publish anything they want, and anyone can comment on those findings. The authors of course have the chance to revise in response to those public comments.

You would lose some of the signaling benefits of journals -- i.e., the editors of publication X thought this met their standards. But there are other ways to achieve the same -- notes of 'I liked this paper' from the same senior authors who would be peer reviewing, for example.

Public posting, public comment has been tried and the results are not good. Authors rarely feel the need to revise in response. The articles tend to be full of spin. The idea seemed decent, at first, but never worked out in practice. Some major OA publishers tried to say that folks needed to read the comments to get context on what may be wrong with the paper; however, many of those platforms did not account for the need to archive the comments along with the PDF so the context and "review" got lost.

Public comment also does not mean that experts with knowledge in that subject area are going to comment. The vast majority of researchers just read the abstract without looking deeper. They certainly do not have time to read the whole article, the comments, or comment on it themselves.

I ran a pilot on this concept for 3 years. The pilot had every article using public posting and commenting. The editors also used traditional peer review. They compared the results and traditional peer review was better and more comprehensive every time.

It's a nice concept and may it work in some fields, but I work on the medical side and it fails as a concept. A doctor who sees patients, teaches and conducts research does not have the time to spend here and the sheer volume of articles will mean that most never seen any comment or review.

 

[Staffan]

Err... wut? @Belen makes a statement. @Staffan responds. I have almost no knowledge in this area, so asked these two to explain their positions, because it seems like an interesting topic. Belen responds with a super cogent, highly detailed response. Staffan responds with what amounts to 'I don't have a firm grasp on this topic either'. I agree with his sentiment, but it looks like he is arguing strictly based on 'feels', barring further explanation. Explain the 'weirdly demanding' aspect to me again?

I believe that if governments are funding research, they should fund the whole process. And I believe governments should fund a whole lot more research. I think a world where all publicly funded research is publicly available would be a better world. As for what steps are needed to get from here to there, I leave that up to the experts.

At a minimum, I believe that if a government grant specifies that any papers published based on that grant has to be open access, the grant should provide the funding to make it so instead of pushing that burden onto the scientists.

 

[The Firebird]

Public posting, public comment has been tried and the results are not good. Authors rarely feel the need to revise in response. The articles tend to be full of spin. The idea seemed decent, at first, but never worked out in practice. Some major OA publishers tried to say that folks needed to read the comments to get context on what may be wrong with the paper; however, many of those platforms did not account for the need to archive the comments along with the PDF so the context and "review" got lost.

Public comment also does not mean that experts with knowledge in that subject area are going to comment. The vast majority of researchers just read the abstract without looking deeper. They certainly do not have time to read the whole article, the comments, or comment on it themselves.

I ran a pilot on this concept for 3 years. The pilot had every article using public posting and commenting. The editors also used traditional peer review. They compared the results and traditional peer review was better and more comprehensive every time.

It's a nice concept and may it work in some fields, but I work on the medical side and it fails as a concept. A doctor who sees patients, teaches and conducts research does not have the time to spend here and the sheer volume of articles will mean that most never seen any comment or review.

I don't have your experience with the trial so I will trust your results there. But in general I think these issues speak more to a mismatch in expectations than failure of concept.

Archiving comments properly is a technical concern and solvable.

As for the readers looking at just the abstract (and, I assume, trusting it) -- frankly, that is an issue even if the research is peer reviewed. The replication crisis, not to mention the outright fraud crisis, is fully exposed at this point. I'm not sure what use case you imagine -- is it doctors reading the abstracts and then treating patients based on this? That would speak to a need for an intermediary between research and clinic (like UpToDate) even with peer review.

Or if it is researchers just reading the abstract, for purposes other than evaluating if they should read further or not...then I am not sure I trust their scholarship.

But yes, under a "public post, public comment" regime, abstracts are going to be less trustworthy. I think this is a good thing, not a bad thing, because it means people stop trusting things they ought not to trust just because it has the peer review stamp.

Another way to phrase it is--peer review is designed to strengthen the "weakest link", the weakest papers. But peer review isn't an effective tool to prevent them, and can make weak links look strong because they have a seal of approval. In a more public regime, the job of vetting and compiling all this information for end users who don't want to read the papers can be done systematically rather than by ad hoc committees.

Unfortunately I am not sure you can capture how the dynamics will shift with a pilot study, because it (hopefully) will lead to a cultural shift in how practicioners engage with scientific papers. But that cultural shift is desirable anyway.

 

[Micah Sweet]

Interesting but odd conversation to be having on an RPG site.

 

[Belen]

Interesting but odd conversation to be having on an RPG site.

How about a nice game of chess?

 

[mcmillan]

Err... wut? @Belen makes a statement. @Staffan responds. I have almost no knowledge in this area, so asked these two to explain their positions, because it seems like an interesting topic. Belen responds with a super cogent, highly detailed response. Staffan responds with what amounts to 'I don't have a firm grasp on this topic either'. I agree with his sentiment, but it looks like he is arguing strictly based on 'feels', barring further explanation. Explain the 'weirdly demanding' aspect to me again?

When Staffan admitted he wasn’t too familiar with academic publishing it seemed clear this wasn’t a conversation of two experts but an expert and someone more familiar than you but still a layperson. You might not have intended this impression but your response to this to ask them be more persuasive and “try again” when they admitted reaching their limit of knowledge is what came off as demanding.

At a minimum, I believe that if a government grant specifies that any papers published based on that grant has to be open access, the grant should provide the funding to make it so instead of pushing that burden onto the scientists

To be clear when we say scientists pay publication fees this almost never is their personal funds (I say almost since I think I remember hearing about some who did cover costs as a political point early on when concepts were first being put forward but currently it would be very unusual). Generally this would come out of grant funds but that would mean a trade off for money that otherwise would go into research. And mandates for OA publishing haven’t kept up with corresponding increase in grant funding. In our department we have had people request publishing support from department budget that would also go to things like conferences or training workshops or there’s smaller funding opportunities the university offers that could offset publication fees (this isn’t what they were created for but is the main use in chemistry where the total grant might cover a few weeks worth of chemicals) Ideally I’d like to see grants cover publishing better too, but in practice there’s a few issues. There’s a range of fees in different journals even in related fields so setting the amount may be tricky. Doing something like saying funding supplement would be given to match publication fees can run into problems with incentivizing more predatory journals to raise fees since they know they’ll be paid. There’s also tricky situations where a grant might be awarded for doing research but not be renewed at the time publication happens. There might be ways to get around some of these hurdles and I think making an effort to address them would be worthwhile in the long term but there aren’t easy answers.

Unfortunately I am not sure you can capture how the dynamics will shift with a pilot study, because it (hopefully) will lead to a cultural shift in how practicioners engage with scientific papers. But that cultural shift is desirable anyway.

I think there are some cultural shifts happening to acknowledge review is more than just formal peer review prior to publication. There are meaningful discussions happening with preprint servers that were mentioned before - though there’s also a dumping of papers that are fundamentally flawed and never changed. Also with my experience with computational chemistry I have seen there are influences from open source software where people are making data in addition to code available in public repositories.

Interesting but odd conversation to be having on an RPG site.

Yeah it has been a tangent - such that I almost forgot which thread this was in. I’ll probably try to avoid derailing more moving forward

 

[Ulorian - Agent of Chaos]

I believe that if governments are funding research, they should fund the whole process. And I believe governments should fund a whole lot more research. I think a world where all publicly funded research is publicly available would be a better world. As for what steps are needed to get from here to there, I leave that up to the experts.

At a minimum, I believe that if a government grant specifies that any papers published based on that grant has to be open access, the grant should provide the funding to make it so instead of pushing that burden onto the scientists.

Like I said, I agree with your sentiment. @Belen had some instructive insights on how the sausage is made. But yes, that is the way it should be, ideally.

My opinion is that all scientific papers/studies should be publicly accessible... as long as they are properly peer reviewed. Stuff that's in the hopper... what's the point of disseminating that? It's got a reasonably high likelihood of being garbage, and what the world does not need is more food for conspiracy theorists.

 

[Ulorian - Agent of Chaos]

When Staffan admitted he wasn’t too familiar with academic publishing it seemed clear this wasn’t a conversation of two experts but an expert and someone more familiar than you but still a layperson. You might not have intended this impression but your response to this to ask them be more persuasive and “try again” when they admitted reaching their limit of knowledge is what came off as demanding.

I see where you're coming from. I was coming at it from the angle of 'if you're going to contradict someone, you'd better have your ducks in a row'. Despite not having expertise in this area, maybe there were other facts informing @Staffan 's opinion?

To be clear when we say scientists pay publication fees this almost never is their personal funds (I say almost since I think I remember hearing about some who did cover costs as a political point early on when concepts were first being put forward but currently it would be very unusual). Generally this would come out of grant funds but that would mean a trade off for money that otherwise would go into research. And mandates for OA publishing haven’t kept up with corresponding increase in grant funding. In our department we have had people request publishing support from department budget that would also go to things like conferences or training workshops or there’s smaller funding opportunities the university offers that could offset publication fees (this isn’t what they were created for but is the main use in chemistry where the total grant might cover a few weeks worth of chemicals) Ideally I’d like to see grants cover publishing better too, but in practice there’s a few issues. There’s a range of fees in different journals even in related fields so setting the amount may be tricky. Doing something like saying funding supplement would be given to match publication fees can run into problems with incentivizing more predatory journals to raise fees since they know they’ll be paid. There’s also tricky situations where a grant might be awarded for doing research but not be renewed at the time publication happens. There might be ways to get around some of these hurdles and I think making an effort to address them would be worthwhile in the long term but there aren’t easy answers.

I think there are some cultural shifts happening to acknowledge review is more than just formal peer review prior to publication. There are meaningful discussions happening with preprint servers that were mentioned before - though there’s also a dumping of papers that are fundamentally flawed and never changed. Also with my experience with computational chemistry I have seen there are influences from open source software where people are making data in addition to code available in public repositories.

Yeah it has been a tangent - such that I almost forgot which thread this was in. I’ll probably try to avoid derailing more moving forward

Thanks for describing your real world experience...