PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress

AI summary

Overview

Research area: Natural Language Processing / AI-for-Science — specifically agentic systems for automated scientific paper assessment and reproducibility.

Technical level: Intermediate. The paper is a systems/agent-architecture paper rather than a deep algorithmic or theoretical contribution. Readers need general familiarity with LLM agents, tool routing, and vision-language models, but no specialized math.

Scope: The paper introduces and evaluates PaperDoctor, an agent framework that produces evidence-grounded, actionable feedback on unfinished scientific manuscripts by combining surface-level screening, per-claim typed verification, and selective re-execution of experiments.

What This Paper Is About

Autoresearch agents and AI-assisted writing tools can now produce full manuscripts at scale, but nothing inside that loop checks whether the claims in those drafts are actually valid — hallucinated citations, overstated novelty, code that contradicts the method section, and unreproducible numbers all flow into the literature. Existing automated reviewers address a different problem: they issue accept/reject verdicts for a venue, which is useful to reviewers but nearly useless to the author who needs to fix the draft. PaperDoctor reframes the task as diagnosis rather than judgment, producing findings modeled on what a human advisor writes in the margin: here is the problem, here is where it is in your paper, and here is what to change.

Key Contributions

1. A holistic, three-level hierarchical pipeline that spends effort proportionally to verification cost. L1 (surface screening) runs on every submission at low cost and covers writing, figures, citation resolution, and claim extraction. L2 (claim verification) routes each extracted claim to a typed verifier that owns its evidence — web search for prior work, a VLM for figures, code analysis for implementation claims, step-by-step derivation for theory. L3 (experiment reproduction) selectively executes experiments in priority order under a compute budget.

2. An evidence-grounded, actionable feedback format. Every finding is a triple: a finding with a reason (Why), a pointer to a specific artifact such as a sentence, equation, code line, page image, or external URL (Where), and a concrete revision suggestion (How). The design guarantees no finding is emitted without both a quote and a suggestion, making each critique independently auditable and directly actionable.

3. Effective experimental reproduction. Rather than only reading the manuscript, PaperDoctor converts experimental claims into executable reproduction plans with numeric targets, ranks them by importance and feasibility, requires manual author approval before running, and judges outcomes against context-sensitive tolerances to surface reproducibility gaps that are invisible from the paper alone.

4. Substantial empirical validation across domains and authorship types. A human study with 25 graduate students across 30 in-progress papers, plus a comparative benchmark of 40 manuscripts spanning machine learning (ICML orals/spotlights), the natural sciences (Nature Communications), the social sciences (Nature Human Behaviour), and AI-written papers (Agents4Science), compared against published referee reports and the Stanford Agentic Reviewer. An interactive demo interface accompanies the release.

Main Findings

• Authors found the feedback broadly useful. All 30 holistic ratings were positive, with a mean of +1.30 (70% "somewhat helpful," 30% "very helpful"). Evidence acceptance across 1,299 items was 70.6% overall, with per-paper rates ranging from 23.5% to 97.0% and a median of 71.1%. Notably, the two authors at the extremes both still rated the review +1, and the correlation between item acceptance and holistic score was positive but not statistically significant (r = +0.29, P = 0.12) — the two measurements capture different things.

• Verifiable claims were accepted far more than stylistic ones. The three L2 claim-verification checks were rejected only 11% of the time, versus 27% for the three L1 surface-screening checks. Experiment design drew the most items (539) with 81% acceptance. Figure review was the weakest check at 42% acceptance, since vision models still misread dense panels and much of figure design is a subjective authorial decision.

• The error/warning split is informative but imperfect. Authors accepted 67% of items labeled Error on evidence and 66% on suggestion, versus 71% of Warnings on both axes. Marking something an Error cut the uncertain share from 11% to 8% but raised rejections from 18% to 25%. Acceptance of evidence almost always implied acceptance of the suggestion (97.1%), showing the two axes are tightly coupled in practice.

• Evidence grounding arises by design, not by chance. Every PaperDoctor finding carries both evidence and a suggestion, versus 1.5% of the agentic reviewer's findings and 35.9% of referee findings. Human referees attach evidence 45.2% of the time; the agentic reviewer only 2.2%. The agentic reviewer's most common pattern (69.9% of findings) is a suggestion with no supporting evidence.

• PaperDoctor produces more feedback with more even coverage. It writes 36.1 to 56.4 findings per reviewer across domains, ahead of the agentic reviewer (25.2 to 44.0) and far ahead of human referee reports (6.7 to 17.5). It spends only 28.4% of findings on the paper's experiments, versus 49.5% for referees and 72.6% for the agentic reviewer, and it is the only source that meaningfully looks outside the paper — 12.5% on external literature and 11.5% on code. It pays for this with length: 50.0 words per finding versus 29.1 for referees.

• Code findings distinguish AI-written from human-written papers. The L2 Code check reports 8.4 findings per paper on AI-written Agents4Science papers against 4.6, 4.1, and 2.4 for the three human-written groups. Experiment design likewise draws the most L2 findings and the widest spread (18.7 for Agents4Science, 16.1 for Nature Science, versus 9.9 and 9.7 for ICML and Social Science).

• Reproduction is bottlenecked twice — before a command runs and after. Of 43.8 experimental claims per paper, only 14.5 become reproduction plans; just 18.0% of those are initially ready to run, and 47.0% never reach a command at all. Once plans do run, 27.0% produce warnings and 34.5% errors.

• Priority ordering correlates with reproducibility. High-priority plans pass 47.3% of the time and error 29.1%; low-priority plans pass 33.6% and error 39.7%. This matches human practice, since important components tend to be prepared and maintained more carefully.

• Training-heavy experiments reproduce worst. Plans requiring training pass only 11.3% of the time and error 58.5%. Inference from released checkpoints passes 39.0% and errors 38.2%. Analysis pipelines pass 48.9% and error 21.4% — the ordering tracks how much of the original setup a run must rebuild.

• The two paper groups exhibit contrasting failure modes. ICML orals/spotlights usually fail after a command runs (132 of 167 plans execute, but only 19 match the reported number, 14.4%). Agents4Science papers usually fail before a command runs (68 of 86 blocked), yet 13 of the 18 that do run match (72.2%).

• Blocked causes are field-specific. Incomplete code stops 65.2% of Social Science plans and 37.1% of ICML's. Nature Science is blocked mostly by wet-lab procedures that cannot be moved to a machine (46.2%). Across all 272 blocked plans, an incomplete runnable environment — missing code or model weights — is the single largest cause at 33.1%.

Methodology in Plain English

The system first parses the inputs once into a shared reusable representation: the paper as section-organized Markdown, the same paper rendered as page images, the code indexed per file, and the parsed bibliography. Each downstream check then requests only the slice it needs, rather than feeding the entire manuscript and repository into a single model call.

Level 1 handles things checkable by browsing. An LLM reads the paper section by section and flags writing problems, quoting the original sentence verbatim as evidence. A vision-language model reviews rendered page images to catch figures that overflow margins, overlap captions, use undersized fonts, or lack contrast. A web-search-backed check resolves each bibliography entry and records a URL only when the backend returns a genuine match, never synthesizing one. Finally, the system densely extracts every verifiable assertion, tags it with evidence types (theory, code, experiments, literature), and dispatches it. All four skills run in parallel.

Level 2 sends each claim to the verifier that owns its evidence. Code-tagged claims are checked against the source — a paper claiming AdamW while the config specifies Adam is flagged as a warning, since the mismatch may be a stale config. Theory claims are re-derived step by step rather than summarized, checking correctness of each step, hidden assumptions, edge cases, and notation consistency. Literature claims are separated into baseline comparisons, cited facts, and novelty assertions, with novelty assertions labeled novel, incremental, or prior_art_exists. Experiment-design claims are reviewed for missing ablations, fairness, statistical rigor, baseline recency, and cherry-picking risk, and each produces a declarative reproduction-plan entry with command, priority, feasibility, run mode, and numeric target. All four verifiers run in parallel.

Level 3 executes selectively. Plans are ranked high/medium/low by importance to central contributions and by expected feasibility, with evaluations preferred over training and ready experiments over blocked ones. Because reproduction consumes real compute and may alter the environment, the plan is presented to the author with estimated GPU hours, dataset size, and storage footprint for approval before an LLM-driven executor handles setup, dispatch, and log parsing. Outcomes are judged in context — considering metric type, reported variance, and the original gap — yielding a pass, warning, or error rather than a fixed numeric threshold.

Evaluation combined a human study (30 in-progress papers from 25 graduate students, 1,299 rated items across evidence and suggestion axes) with a comparative benchmark (40 papers from four groups, each carrying published referee reports, one run of the Stanford Agentic Reviewer, and one PaperDoctor run). Because the three sources have different output structures, Gemini-2.5-Flash was used to parse all feedback into a common set of atomic points for fair comparison.

Why This Matters

The paper reframes automated paper assessment from a verdict machine into a diagnostic instrument, and backs that reframing with measurements showing the two produce genuinely different artifacts. The comparison is concrete: a reviewer-style agent optimizes for agreeing with held-out reviewer opinions, while PaperDoctor optimizes for whether authors accept and act on each finding. Given that most human drafts are now at least partly AI-assisted and autoresearch pipelines have no internal quality-control stage, a scalable pre-submission diagnostic fills a gap that neither human advisors (too slow) nor existing reviewers (wrong output contract) can cover.

Real-world applications:

• Pre-submission self-check for research groups. Authors can run a draft plus its repository through the pipeline before submitting, catching citation hallucinations, paper-code mismatches, missing ablations, and unreproducible numbers while there is still time to fix them.
• Advisor and mentor scaling. A single advisor can supervise more students when routine checkable issues — typos, unresolvable references, figure margin overflows, undefined notation — are surfaced automatically and clearly located.
• Reproducibility auditing for venues and journals. The L3 stage demonstrates a tractable protocol for selectively re-executing claims under a compute budget with author approval, which could inform artifact-evaluation workflows or post-publication integrity checks.
• Quality gating inside autoresearch pipelines. Because PaperDoctor consumes a draft rather than producing one, it can be inserted as the missing internal quality-control stage in existing end-to-end paper-generation systems.

Industry relevance: Organizations deploying LLMs for technical writing, documentation, or research reports face the same class of failure — claims in prose that contradict the code or data behind them. The typed-verifier architecture, the evidence-grounded output contract, and the cost-tiered pipeline design are all transferable to any domain where generated text must be checked against executable artifacts. The paper's finding that AI-written papers exhibit a distinct code-mismatch signature (8.4 findings per paper versus 2.4 to 4.6 for human-written work) is directly relevant to teams shipping AI-assisted technical content.

Future Directions

• Interactive accept/contest loop. The authors propose letting authors accept or contest each finding, with the affected checks re-running in response. This would turn the one-shot report into a dialogue and could sharpen precision over successive rounds.

• Improving figure and visual assessment. Figure review had the lowest acceptance rate (42%) of any check, and the authors attribute this to VLM limitations on dense panels plus the inherent subjectivity of figure design. Better visual grounding is the clearest single-check improvement available.

• Judging novelty and framing. Among the 24 free-form comments, three aspects drew more criticism than praise: the proposed fixes, the items PaperDoctor got wrong, and its silence on novelty and framing. The authors acknowledge that these require judging what is worth saying rather than checking whether something is true, and leave them as human calls — but users clearly want more.

• Tackling the reproduction bottleneck. Nearly half of all reproduction plans (47.0%) never reach a command, most often because repositories lack enough information to determine how to run them, and 33.1% of blocked plans suffer from an incomplete runnable environment. Training-heavy plans pass only 11.3% of the time. Reducing this pre-execution blockage — through better environment reconstruction or community standards for runnable artifacts — is where the largest gains in verification coverage lie.

Target Audience

Most valuable for: researchers actively writing papers (particularly graduate students and junior researchers who lack frequent advisor access), research advisors and mentors seeking to scale their feedback, and NLP/ML researchers building agentic review or AI-for-science systems.

Also relevant for: conference and journal organizers evaluating artifact-checking and reproducibility workflows, AI ethics and research-integrity researchers concerned about AI-generated content entering the literature, and industry teams deploying LLMs for technical writing where generated prose must be validated against underlying code or data.

Less relevant for: readers seeking theoretical contributions, novel learning algorithms, or deep architectural innovation — the contribution here is a systems design and a thorough empirical characterization of how automated diagnostic feedback compares to human and agentic reviewing.

Authors’ abstract

Autoresearch agents are reshaping the research ecosystem, but they can also let flawed claims enter the literature at scale. Human advisors catch such issues in drafts through careful, traceable feedback, yet advisor-style assessment requires extensive manual effort and does not scale. To shift automated paper assessment from a judge to a diagnostician, we introduce PaperDoctor, an agent framework for pre-submission feedback with three key innovations. First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments through three layers: L1 surface screening, L2 typed verifiers that route each claim to the appropriate evidence, and L3 reproducers that rerun experiments by priority. Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion, making critiques auditable and actionable. Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget, surfacing reproducibility gaps and quantitative limitations that are invisible from the manuscript alone. We evaluate PaperDoctor on 30 in-progress papers, yielding 70.6% agreement and all positive holistic scores, and on 40 manuscripts across machine learning, natural science, and social science, covering human- and AI-authored papers with code. Overall, PaperDoctor produces more auditable feedback than human and other agentic reviewers, pairs critiques with concrete suggestions by design, and complements dimensions often overlooked by human reviewers. We also develop an interactive interface that lets authors browse findings grounded in their paper. PaperDoctor reframes automated paper assessment as diagnosis rather than verdict, taking a concrete step toward AI advisors for more rigorous AI-assisted scientific discovery.