FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers

AI summary

Overview

Research area: Natural Language Processing / evaluation of Large Language Models for scientific peer review (error detection and fault localization).

Technical level: Intermediate. The paper's framing (benchmark construction, error insertion, LLM-as-a-judge evaluation) is accessible to readers familiar with basic LLM evaluation concepts; the formal metric definitions require some comfort with text-similarity measures.

Scope in one sentence: The paper builds FLAWS, an automated benchmark of 713 paper-error pairs created by inserting claim-invalidating errors into peer-reviewed papers, and uses it to measure how well five frontier LLMs can identify and localize those errors.

What This Paper Is About

Peer review depends on reviewers catching errors that undermine a paper's central claims, but the rapid growth of scientific output far outstrips the limited pool of qualified experts, making reliable error detection harder. The authors ask whether LLMs can help by pinpointing erroneous text in manuscripts, and they note that existing review-quality evaluations do not test this specific identify-and-locate ability. To answer it, they construct a scalable benchmark in which controlled, paper-specific errors are injected into real peer-reviewed papers, and they score models on whether they surface the right passages.

Key Contributions

1. A framework for controlled error insertion. Rather than drawing from a fixed catalog of error types, the pipeline extracts a paper's own falsifiable claims and inserts an error that invalidates one of them, which makes errors relevant to the paper's substance and applicable to any paper, including theoretical and survey papers. The pipeline also filters out invalid and trivial errors.
2. An automated benchmark and evaluation metric. FLAWS contains 713 paper-error pairs drawn from 448 unique papers with errors inserted by Gemini 2.5 Pro and 265 unique papers with errors inserted by GPT 5, paired with a word-level Levenshtein-based similarity metric plus an LLM-as-a-judge metric, so evaluation does not require expensive manual annotation.
3. An evaluation and ranking of five frontier LLMs. The paper evaluates Claude Sonnet 4.5, DeepSeek Reasoner v3.1, Gemini 2.5 Pro, GPT 5, and Grok 4, and reports a ranked table of identification accuracy.
4. A release of code and data at the project's GitHub repository.

Main Findings

• GPT 5 is the top-performing model. It achieves 39.1% identification accuracy at k = 10, where k is the number of top-ranked error text candidates generated by the model, and 19.2% at k = 3. It receives a score (β_j) of 2.10 and is ranked 1.
• The full ranking on FLAWS (accuracy at k = 3 / k = 10): GPT 5, rank 1, β_j = 2.10, 19.2% / 39.1%; DeepSeek Reasoner v3.1, rank 2, β_j = 1.90, 16.3% / 35.2%; Grok 4, rank 3, β_j = 1.68, 16.3% / 23.4%; Claude Sonnet 4.5, rank 4, β_j = 1.47, 12.6% / 21.5%; Gemini 2.5 Pro, rank 5, β_j = 1.41, 15.7% / 19.8%.
• Accuracy rises with k for every model, but absolute performance remains far from complete detection, with the top model identifying roughly two-fifths of inserted errors at k = 10.
• Ranking by accuracy alone can mislead. The authors state they compute the score β_j from the identification results to control for confounding factors, and that the final ranking uses β_j. The paper's ranking uses β_j from Table 1, though Table 1 also reports the raw accuracies. (Grok 4 and DeepSeek Reasoner v3.1 both have 16.3% at k = 3 but differ at k = 10, and Gemini 2.5 Pro has a higher k = 3 accuracy than Claude Sonnet 4.5 yet a lower k = 10 accuracy — the reasons are addressed in Section 5.2, whose contents are not included in the provided text.)
• Filtering removed most generated errors. From 623 source papers, Gemini 2.5 Pro produced 4,025 errors and GPT 5 produced 9,154. After filtering invalid errors, 4,008 (99%) and 9,105 (99%) remained. After filtering easy errors, 1,560 errors (39%) across 526 papers (84%) remained for Gemini 2.5 Pro, and 632 errors (7%) across 341 papers (58%) remained for GPT 5. After PDF compilation, 1,287 errors (32%) across 448 papers (72%) remained for Gemini 2.5 Pro, and 468 errors (5%) across 265 papers (43%) remained for GPT 5.
• The benchmark is grounded in recent, peer-reviewed papers. Papers were accepted to ICML 2025 and first posted on arXiv after January 2025, chosen to reduce the risk of data contamination; 623 papers with LaTeX sources were downloaded on October 9, 2025.
• Reliability was checked against human judgment. Validation used 29 randomly selected papers appearing in both the Gemini 2.5 Pro- and GPT 5-inserted subsets, with manual judgments collected for all five identification models and three graduate-level ML researcher annotators, with each paper independently judged. (The remaining validation results are truncated in the provided text.)
• Model knowledge cutoffs vary: DeepSeek Reasoner v3.1 (July 2024), GPT 5 (September 30, 2024), Grok 4 (November 2024), Gemini 2.5 Pro (January 2025), and Claude Sonnet 4.5 (March 2025). The authors note Claude Sonnet 3.7 (cutoff November 2024) was tried first but frequently failed to detect any errors, so Claude Sonnet 4.5 was used instead.

Methodology in Plain English

Building the benchmark. The authors start from 623 ICML 2025 papers with available LaTeX sources. For each paper, an LLM reads the source and lists the paper's key falsifiable claims. For each claim, an LLM is then asked to modify one or more passages so that the claim no longer holds, producing a substantive conceptual flaw rather than a surface-level stylistic slip. The model outputs the modified text, the original text, and an explanation. It may edit several places to plant a single error.

Filtering. Two filters remove unsuitable errors. First, a fresh LLM instance checks that the modified text does not announce the error and does not merely resurface a pre-existing limitation. Second, errors that are too easy are removed: an LLM judges whether an error is trivially easy, and separately the inserting model itself tries to identify the error from the modified source ("internal identification"). Because an inserted error can affect passages beyond the edited ones, a separate step asks the model to return all passages related to the error, and that combined set becomes the ground truth. Any error the model finds is discarded. Identification here is judged by word-level Levenshtein similarity above 0.5.

Insertion mechanics. The original excerpt is located in the LaTeX by exact match; when that fails, difflib.SequenceMatcher finds the closest span, and the replacement is made only if similarity exceeds 0.9 to guard against hallucinated text. Modified LaTeX is compiled to PDF with pdflatex so models are evaluated on PDFs as human reviewers would read them; pairs that fail to compile are discarded, and one modified PDF is randomly sampled per paper.

Evaluation. Each model receives the full PDF and is explicitly asked to return erroneous excerpts ranked by significance, not to write a holistic review. Two anti-gaming constraints apply: at most 10 excerpts, and each excerpt no longer than the corresponding ground truth error excerpt. An error counts as identified if it is flagged by either the word-level Levenshtein metric (threshold above 0.5, using sentence-level sub-span matching) or the LLM-as-a-judge metric. Using two metrics makes the external evaluation more lenient than the internal identification filter.

Why This Matters

Impact on research. The paper isolates and measures a specific, consequential peer-review subtask — finding and locating errors — rather than overall review quality, which prior work showed is poorly served by holistic "review this paper" prompts. It offers a scalable, generalizable alternative to benchmarks built from retraction notes or from manual annotation, and it supplies a ranked baseline for comparing models.

Real-world applications:

• Assisting human reviewers by surfacing candidate error passages so experts can focus verification effort where it matters most.
• Pre-submission self-checking by authors who want likely problems flagged before review.
• Screening pipelines at conferences and journals, where the volume of submissions outpaces the reviewer pool.
• Post-publication checks or editorial triage that flag manuscripts needing closer scrutiny.

Industry relevance. The work is relevant to vendors and deployers of LLMs, to publishers and conference organizers evaluating whether to incorporate AI review assistance (the paper cites AAAI 2026 incorporating LLM-generated reviews), and to anyone building evaluation harnesses for long-document scientific reasoning. The benchmark and code are released publicly.

Future Directions

• Improve identification and localization accuracy. The best model reaches 39.1% at k = 10, leaving substantial room for better methods, prompting, or fine-tuning.
• Investigate why the models rank as they do. The paper states that Section 5.2 analyzes the identification results and that the β_j score controls for confounding factors; the full analysis and the derivation of β_j are not contained in the provided text and would be a natural point of scrutiny.
• Reduce construction cost and repeat filtering. The authors note the filtration pipeline is very costly, was run only once, and that repeating it could remove additional easy errors due to the stochastic nature of LLM generation.
• Compare insertion models and error properties more fully. FLAWS deliberately uses two insertion models (Gemini 2.5 Pro and GPT 5) and filters out errors detectable by the inserting model, but the paper notes only that evidence of a strict, consistent distinction between insertion-model subsets is not reported at length in the provided content, and the appendices referenced for prompts and examples are not included here.

Target Audience

Researchers and practitioners in NLP and AI evaluation, machine learning conference reviewers and organizers, and anyone building or evaluating automated peer-review or scientific-validity tools. It is most useful to readers who want a rigorous, reproducible benchmark for model error-detection ability, and to those assessing whether LLMs are ready to assist — rather than replace — human review.

Authors’ abstract

The identification and localization of errors is a core task in peer review, yet the exponential growth of scientific output has made it increasingly difficult for human reviewers to reliably detect errors given the limited pool of experts. Recent advances in Large Language Models (LLMs) have sparked interest in their potential to support such evaluation tasks, from academic peer review to automated scientific assessment. However, despite the growing use of LLMs in review systems, their capabilities to pinpoint errors remain underexplored. In this work, we introduce Fault Localization Across Writing in Science (FLAWS), an automated benchmark consisting of 713 paper-error pairs designed to evaluate how effectively LLMs detect errors that undermine key claims in research papers. We construct the benchmark by systematically inserting claim-invalidating errors into peer-reviewed papers using LLMs, paired with an automated evaluation metric that measures whether models can identify and localize these errors. Developing such a benchmark presents unique challenges that we overcome: ensuring that the inserted errors are well-defined, challenging, and relevant to the content of the paper, avoiding artifacts that would make identification trivial, and designing a scalable, automated evaluation metric. On the resulting benchmark, we evaluate five frontier LLMs: Claude Sonnet 4.5, DeepSeek Reasoner v3.1, Gemini 2.5 Pro, GPT 5, and Grok 4. Among these, GPT 5 is the top-performing model, achieving 39.1% identification accuracy when k=10, where k is the number of top-ranked error text candidates generated by the LLM.