Sakana AI's AI Review Catches 73% of Planted Errors — but Only 16% in Retracted Papers

Sakana AI's AI Review Catches 73% of Planted Errors — but Only 16% in Retracted Papers

Sakana AI's new peer review system impresses on its own benchmark, where it finds 73.43% of core-claim errors at roughly $0.47 per review. But on genuinely retracted papers, accuracy falls to 16.11% — and that gap is what will decide whether AI can help peer review in practice.

Sakana AI has published the research paper "Beyond Imitation" in TMLR, describing a system for AI-assisted peer review built around error detection. The paper presents two things at once: a benchmark of its own for measuring whether language models actually find errors in papers, and a review system called Multi-Layered Review (MLR) designed to beat existing baselines on that benchmark.

The numbers come from MarkTechPost's write-up of the paper (October 10, 2026); AIMag has not independently verified the TMLR paper itself, and all results below are therefore reported secondhand.

How MLR Works

Multi-Layered Review is an agent-based system built on standard Claude models — Sonnet 4 and Haiku 3.5. Three agents are tasked with understanding a research paper before criticizing it, and the work is organized as a three-phase prompt chain inspired by Keshav's well-known "Three-Pass Approach" for reading scientific papers. The cost, per MarkTechPost, is about $0.47 per completed review.

This is not a new model, but an orchestration of existing models with a reading strategy. The available coverage does not describe in detail what each of the three phases concretely does in the system.

The Benchmark: Contradictions Planted by Recipe

To be able to measure error detection, the team built a "Contradiction Benchmark." They collected 257 CC-licensed papers from ACL, AISTATS, CVPR and ICML 2025, plus NeurIPS 2024. Gemini 2.5 Pro then built a knowledge graph of each paper — claims, evidence, and methods — and GPT-4.1 rewrote one node per graph distance into a contradiction. The distance from the paper's "main claim" determines severity: distance 0 strikes at the core of the paper, larger distances hit details. In this way, 257 papers become 1,164 inserted contradictions.

The reviews are scored by an o3 judge that scores each review ten times. On clean papers the judge reached 99.9% accuracy, and on manually confirmed findings it showed 86.8% sensitivity — which, according to MarkTechPost, suggests the reported detection numbers may be conservative, i.e., more likely too low than too high.

The Results — and What They Don't Show

With four reviews, MLR caught, per MarkTechPost's summary, 73.43% of the contradictions at distance 0 (core claims) and 40.95% overall. The best baseline, AgentReview, caught 14.81% at distance 0. MarkTechPost thus reports a clear margin over the strongest baseline, but complete results for all baselines are not reproduced in the available coverage.

On ICLR 2025 submissions, MLR's predicted score correlated with human reviewers' scores with a Pearson correlation of 0.586 — against a reference point of 0.742 for the correlation between human reviewers themselves. The system is thus not at the level of human agreement, but not far off either.

The essential caveat comes on real-world data: on 211 retracted arXiv papers from the WithdrarXiv-Check dataset, the lead shrinks considerably. MLR scored 26.07% on "similar" hits and 16.11% on "exact" hits, against 18.48% and 9.00% for the strongest baselines. The lead is there, but the absolute level is low: the system does not find most of the errors that actually led to retraction.

The explanation may lie in how the two test sets differ. The benchmark errors are constructed according to a deterministic recipe — a knowledge graph, a node rewritten into a contradiction, severity measured in graph distance. Such errors have a structure that an agent reading in phases can learn to look for. The errors in retracted papers, by contrast, arose in the wild: inconsistent data, overblown conclusions, methodological problems. The drop in hit rate from 73.43% to 16.11% on exact hits is a direct expression of that gap.

Why This Lands Now

The debate over whether AI-generated research can be trusted is in full swing. On October 8, 2026, TechCrunch covered the paper "lost in translation," in which mathematicians argue that OpenAI's natural-language proofs and other auto-formalized Lean proofs should not be trusted at face value but must go through the same peer review process and scrutiny as other proofs. Sakana AI's MLR can be read as one answer to precisely that trust problem: if AI generates more research, tools are needed that can relieve — not replace — human peer review.

MarkTechPost's summary also reports that the system remains vulnerable to hidden prompt injection — that is, malicious instructions hidden in a paper can manipulate the review. This point is underdocumented in the available coverage, and the extent of the vulnerability cannot be verified without the paper itself.

What Remains to Be Answered

The open questions are concrete. Can MLR's approach transfer from planted contradictions to real error types, or would entirely different training and reading strategies be needed? How representative is the 0.586 ICLR correlation across fields and conferences? And how serious is the prompt-injection vulnerability in practice, given that manipulated papers are a known attack pattern against automated reviewers?

For now, the evidence supports one conclusion: MLR is a promising assistant for error detection with clearly measurable strengths — and a benchmark performance that has not yet been demonstrated on the errors that matter most.

Sources: MarkTechPost (October 10, 2026), TechCrunch (October 8, 2026).