The Pulse
Sakana AI’s Review System Finds 73% of Planted Core Errors
Sakana AI’s Multi-Layered Review system detected 73.43% of planted contradictions tied to papers’ central claims when using four reviews. On 211 withdrawn arXiv papers, it led the listed systems in both similar-match and exact-match rates,
AI.info Team · Written with AI ·
With four reviews, Sakana AI’s Multi-Layered Review system detected 73.43% of contradictions planted in the central claims of research papers. The result comes from a purpose-built test, not a routine assessment of untouched submissions: across the full benchmark, MLR caught 40.95% of inserted contradictions. On a separate set of withdrawn arXiv papers, its exact-match rate was 16.11%.
The figures come from “Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review,” which evaluates automated reviewers by whether they find errors, rather than how closely their written reviews resemble human ones. The paper’s authors write:
“Here, we introduce a verification-centric perspective on LLM-assisted peer review, emphasizing error detection as a critical and resource-intensive task.”
Four reviews catch more planted contradictions
The authors built the Contradiction Benchmark from 257 papers and inserted 1,164 contradictions into them. The source papers came from ACL, AISTATS, CVPR and ICML in 2025, as well as NeurIPS in 2024. The inserted errors were ranked by how closely they related to a paper’s main claim, making contradictions at distance zero the benchmark’s most consequential category.
MLR’s four-review setup detected 73.43% of those distance-zero contradictions. The strongest baseline, AgentReview, detected 14.81%; MLR’s single-review configuration reached 60.79%. Across contradictions at all severity levels, however, the four-review score fell to 40.95%, a reminder that the headline result applies to one particularly central class of synthetic errors.
MLR reads before it writes
The system divides review work among agents and stages. An Appendix Agent summarizes experimental and implementation details; an optional Literature Review Agent searches for related work; and a Review Agent first outlines the paper, then examines it for weaknesses and evidence, and finally assembles its findings into a review. In the reported configuration, the agents use Claude Haiku 3.5 and Claude Sonnet 4, according to MarkTechPost’s account of the paper.
The study also separates gains from model choice and system design. In one comparison, replacing GPT-4.1 with Claude Sonnet 4 in the LLM-Review baseline raised its detection of distance-zero contradictions from 14.56% to 35.40%, still below MLR’s reported result. MarkTechPost reports a cost of about $0.47 per MLR review, excluding the optional literature-search agent.
Retracted papers cut the score to 16.11%
Controlled contradictions are easier to score than flaws documented in real research. On WithdrarXiv-Check, a dataset of 211 withdrawn arXiv papers, MLR’s rate was 26.07% when a reviewer’s finding counted as similar to the documented problem, and 16.11% under exact matching. MLR led both measures among the listed systems: AgentReview, the strongest baseline on similar matches, reached 18.48%, while the highest baseline exact-match rate was 9.00%.
That gap matters for researchers considering automated review as a screening aid. The benchmark shows that a staged system can identify many deliberately inserted errors tied directly to a paper’s main claim; the withdrawn-paper test shows a much lower hit rate against recorded real-world problems. The paper also reports that MLR’s predicted scores on ICLR 2025 submissions correlated with human scores at 0.586, below the reported human-to-human reference of 0.742.
A strong benchmark result, not a substitute for review
MLR’s results support a narrower claim than “AI can check papers”: on this constructed benchmark, a multi-stage reviewer outperformed three tested automated baselines, especially on contradictions affecting central claims. Its lower exact-match rate on withdrawn papers limits what the synthetic score can establish about ordinary peer review. The authors frame automated reviewing as a source of complementary feedback, rather than a replacement for human reviewers.
The practical questions are specific: how well does MLR identify flaws in new, unaltered submissions, and how much does its detection change when a paper contains instructions designed to manipulate an AI reviewer? The 73.43% score answers neither on its own.