__  __      _        _____ _ _ _
|  \/  | ___| |_ __ _|  ___(_) | |_ ___ _ __
| |\/| |/ _ \ __/ _` | |_  | | | __/ _ \ '__|
| |  | |  __/ || (_| |  _| | | | ||  __/ |
|_|  |_|\___|\__\__,_|_|   |_|_|\__\___|_|
community weblog	

Peer Review will never be the same.

We show that AI-generated article reviews are now better, by every measure we could construct, than reviews written by well-compensated human experts and judged to be of unusually high quality by journal editors and authors.

Overall assessment: This is a compelling empirical result, supported by a reasonably sophisticated experimental design, but the paper's headline claim is stronger than its evidence warrants.

The authors provide substantial evidence that frontier AI systems can produce scholarly manuscript reviews that are preferred to unusually well-regarded human reviews. However, they do not establish that those reviews produce better science, detect the most consequential errors more reliably, or can safely replace human peer review.

My central distinction is between the quality of a review as judged by an evaluator and the scientific value of the review's consequences. The paper measures the former, not the latter.

...

3. The most important methodological weaknesses
The outcome variable is a proxy, not scientific improvement

The researchers ask evaluators which review is better. They do not measure what happens when authors implement the feedback.

A review that identifies 20 legitimate minor issues might be preferred to one identifying a single fundamental flaw in a causal identification strategy. Yet the second could be far more valuable scientifically.

The paper does not establish that its evaluation rubric weights these outcomes appropriately.

This is the principal threat to construct validity. The authors acknowledge that they did not measure subsequent manuscript improvements, but this limitation directly constrains their main conclusion...

The human validation is small and statistically dependent

The headline human result is 26 preferences out of 30. But these are not 30 independent manuscripts: three editors each assessed the same ten pairs.

A naive binomial analysis would overstate the effective sample size because judgments can be correlated within manuscripts and editors.

The authors could strengthen the evidence with more manuscripts, independent editorial panels, and hierarchical statistical analysis.

The result is promising, but the human validation is much narrower than the 134-pair AI evaluation suggests...

Generalization is limited

The sample comes from a particular academic review service, with a substantial social-science orientation and a constructive-feedback mandate.

It does not establish comparable performance for laboratory protocols, clinical research, specialized mathematical proofs, or fields where expert judgment depends on tacit practical knowledge.

Nor does it establish that AI can evaluate novelty or publication worthiness, tasks explicitly excluded from the service's main instructions...

How rapidPeer is funded
rapidPeer is free to authors, and reviewers receive an honorarium for their work. Both are made possible by a generous gift from Alexander and Diviya Magaro.

Peer review doesn't have to be all human or all AI. You could do both. If you're a human reviewer, you could use AI to help you do your human review.

I think that AI is a problem not because it challenges our notion of specialness but because it has the potential to be a deathmatch between top-down systems (authoritarianism, unrestrainable capitalism) and the flourishing of the majority.

Seeing technologies as instruments of political as well as material production complicates a familiar argument from the early days of science and technology studies, namely, in the words of political scientist Langdon Winner, that 'artifacts have politics' (Winner 1980). Winner suggested that the very design of technological systems, such as their relative opacity or transparency, aligns with particular forms of politics.

A highly black-boxed and inscrutable technological system, such as nuclear power generation [insert AI here], lacks transparency, Winner argued, and thus deprives citizens of meaningful opportunities to take part in managing it. The same could be said of something as simple as a highway design that lets some vehicles freely get to some locations but inhibits other forms and routes of travel.

Technology's unchecked power over our lives, exercised through often invisible design choices, is inconsistent with democracy's first principle: That members of a polity must have opportunities to penetrate the workings of power, demand explanations, and say yea or nay to the choices made by governing political institutions. Anything else would be plainly anti-democratic. (Jasanoff 2020, p. 31-32)

With the growth of scientific output, the peer review system is approaching a breaking point: reviewers are increasingly hard to secure, and turnaround times lengthen accordingly. LLM-assisted or LLM-generated peer review has been proposed as a remedy, but how such models would treat manuscripts already published in top-tier journals, the closest thing the current system has to a gold standard of scientific validity, is not known. Here we assembled a corpus of 200 articles, each released under a CC BY 4.0 license and published in 2026, from 20 general-science, life-science, and clinical-medicine journals including Nature, Science, and Cell, and asked four current LLMs (two proprietary, two open-weight) to produce independent peer-review reports of each one under a blinded protocol. Their decisions were overwhelmingly harsh and starkly inconsistent with each other: 605 of 800 reviews (75.6%) recommended major revision or rejection, but the rate at which any single model did so ranged from 17.5% to 100% (p < 0.001), disagreement on manuscripts that had, by construction, already survived real editorial review at the journals in question. When the same manuscripts were doctored with one deliberately planted, field-specific methodological or statistical flaw, detection rates varied from 17.0% to 60.5% across models (p < 0.001). Recommending revision for papers that had already passed editorial review (low specificity) while missing flaws deliberately placed in front of them (low sensitivity), these models produce output that should be read for what it is: an uneven, framing- and model-dependent signal.