Peer review

Peer review

Here you will find papers and resources related to the study of peer review.

Papers on peer review

Is gatekeeping necessary?

• Anything Goes? Why Uphold Peer Review at All | by Ann-Sophie Barwich | Mar, 2024 | Medium
• Heesen, R., & Bright, L. K. (2021). Is Peer Review a Good Idea? The British Journal for the Philosophy of Science, 72(3), 635-663. The answer is no (Goodwin's law of headlines).
• Desmond, H. (2024). Gatekeeping should be conserved in the open science era. Synthese, 203(5), 160

Studying Peer Review

• Revise and Resubmit: An Intertextual Model of Text-Based Collaboration in Peer Review
• Does My Rebuttal Matter? Insights from a Major NLP Conference
• Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond
• A Corpus Study of the Term Evidence in Open Peer Reviews to Research Articles in the British Medical Journal
• Open Peer Review Urgently Requires Evidence: A Call to Action | PLOS Biology

Questionable Reviewing Practices

• Reviewer-Coerced Citation: Case Report, Update on Journal Policy and Suggestions for Future Prevention
• Review Mill at MDPI
• Unpacking P-Hacking and Publication Bias - American Economic Association
• The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates A study on the use of AI-assisted peer reviews in the 2024 International Conference on Learning Representations (ICLR) found that at least 15.8% of reviews were written with AI assistance, and that AI-assisted reviews were more likely to score higher than human reviews and increase the chances of submission acceptance.

Biases and Unintended Consequences of Peer Review

• Is Blinded Review Enough? Blinded review aims to reduce bias and increase diversity in scientific research. However, a study of Gates Foundation grant proposals found that female applicants received lower scores despite blinded review. This disparity is linked to gender differences in communication styles, not reviewer bias or proposal quality. Female applicants showed greater scientific output after an accepted proposal, highlighting the need to address gender disparities in evaluation.
• Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation
• Western scientists more likely to get rejected papers published — and do it faster – a Nature piece on the preprint Geographical Disparities in Navigating Rejection in Science Drive Disparities in its File Drawer
• Research funding: past performance is a stronger predictor of future scientific output than reviewer scores

Peer Review Costs

• Time to Rethink Academic Publishing: The Peer Reviewer Crisis | mBio

Innovations

• Using Natural Language Processing to Support Peer-Feedback in the Age of Artificial Intelligence: A Cross-Disciplinary Framework and a Research Agenda

• Machine Learning in Scientific Grant Review: Algorithmically Predicting Project Efficiency in High Energy Physics

• Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical Analysis

• Ending Human-Dependent Peer Review - The Scholarly Kitchen

• Researchers Use GPT-4 to Generate Feedback on Scientific Manuscripts

• Code Review Automation: Strengths and Weaknesses of the State of the Art

• Can large language models provide useful feedback on research papers? A large-scale empirical analysis

  GPT-4 can generate scientific feedback on research papers with quality comparable to human peer reviewers:

  • GPT-4's feedback overlaps with human reviewers at similar rates as human-to-human reviewer overlap.
  • Over half of researchers (57.4%) found GPT-4 feedback helpful/very helpful.
  • 82.4% of researchers considered GPT-4 feedback more beneficial than some human reviewers.
  • GPT-4 shows potential in addressing challenges in obtaining timely, high-quality peer reviews, especially for junior researchers or those from under-resourced settings

Opening Peer Review

• The Peer Reviewers' Openness Initiative: incentivizing open research practices through peer review
• Peer Community In - Free Peer Review & Validation of Preprints of Articles
• Recommendations for Accelerating Open Preprint Peer Review to Improve the Culture of Science | PLOS Biology

Peer Review of Code

• Seamless Sharing and Peer Review of Code | Nature Computational Science
• Reproducible Research in Mathematical Sciences Requires Changes in Our Peer Review Culture and Modernization of Our Current Publication Approach - PMC
• Trust but Verify: How to Leverage Policies, Workflows, and Infrastructure to Ensure Computational Reproducibility in Publication, Issue 2.4, Fall 2020

Datasets and open peer review journals

Corpora

• Peer Review Innovations (Survey Results) Very useful summary of various innovations in the peer review process.
• eLife Open Peer Review Corpus
• PLoS Open Peer Review Corpus
• MDPI Open Peer Review Corpus
• MDPI Open Peer Review Corpus 2 (much bigger)

Publishers and Journals

• Meta-Psychology Provides negative reviews as well.

Symposia