ActReview: Rebuttal-Guided Training Data and Rubric Rewards for Actionable Peer Review Generation

AI summary

Overview

Research area: Natural Language Processing, specifically LLM-based scientific peer-review assistance, post-training for open-ended (non-verifiable) generation, and evaluation benchmark construction.

Technical level: Intermediate. The framing (diagnose a weakness, then suggest a revision) is intuitive, but full appreciation of the method requires familiarity with supervised fine-tuning, reinforcement learning from relative feedback, and LLM-as-judge evaluation.

Scope: The paper introduces a dual-task formulation of "actionable peer review," a 40K-instance training set built from real review–rebuttal threads, a 1,000-instance human-curated benchmark, and a post-training recipe that teaches an 8B-parameter model to produce grounded, revision-oriented feedback.

What This Paper Is About

Most LLM review assistants can tell an author what is wrong with a paper, but not what to do about it — they produce descriptive criticism rather than prescriptive, implementable guidance. This paper reframes the task as Actionable Peer-review Generation, split into two explicit subtasks: generating diagnostic claims that pin down a specific paper deficiency, and generating suggestions that say what to revise, where, how, and to what expected effect. The core insight is that author rebuttals on OpenReview already contain evidence of what actions address reviewer concerns, so they can serve as latent supervision for teaching models to produce revision guidance.

Key Contributions

1. A dual-task formulation of actionable review. The paper separates weakness diagnosis from revision planning, with weakness-conditioned inference that allows the model to abstain when the target concern is not supported by the paper. A controlled comparison shows this two-task design outperforms joint end-to-end generation on suggestion quality and claim–suggestion alignment.

2. ActReview-40K, a rebuttal-guided training corpus. Built from 15,819 papers across ICLR, NeurIPS, and EMNLP, yielding roughly 40,000 weakness–response instances. Each instance passes through atomic weakness extraction, weakness–rebuttal alignment, rebuttal-guided rewriting into initial-review-style feedback, and localized evidence retrieval. Rebuttals are never shown to the model — they only shape the supervision offline.

3. ActReview-Bench, a human-curated evaluation benchmark. 1,000 instances retained from 2,000 candidates after double annotation and adjudication, with 0.94 F1 on weakness–rebuttal alignment and 0.89 F1 on instance filtering. Rebuttal-derived actions are treated as grounded reference signals rather than unique gold answers.

4. A post-training recipe with candidate-aware rubric rewards. Multi-task SFT on Qwen3-8B-Base followed by GRPO using instance-specific rubrics constructed from human-written references, model-generated references, and sampled candidate failures. This outperforms direct LLM-judge rewards and fixed task-level rubrics.

Main Findings

• Strong gains over specialized review models. ActReview-RL outperforms DeepReviewer-14B, OpenReviewer-8B, RbtAct-8B, and Qwen3-32B on actionability and grounding dimensions, and is preferred in pairwise human and LLM-judge comparison, particularly on the suggestion task.

• Competitive with frontier proprietary LLMs despite an order-of-magnitude parameter gap. Against GPT-5.1 and Gemini-3.1-Pro-Preview, preferences are close to parity, though schema-controlled prompting variants close part of the gap — output structure alone explains some of the advantage.

• Technical accuracy remains the weak point. Across both LLM-judge and human evaluation, Technical Accuracy is the dimension where frontier models stay most competitive. The authors explicitly state their results should not be read as universal technical superiority; gains concentrate on specificity, depth, and actionability.

• Rebuttal enhancement drives the largest improvements. Ablation shows replacing raw review data with rebuttal-enhanced data raises Task 2 rubric score by 0.12 and evidence specificity by 1.73 points, far larger than any other single component.

• Instance-specific rubrics beat generic ones. Candidate-aware rubrics (Task 2 Overall 73.4) outperform direct judge reward (68.7), fixed task-level rubric (61.0), and SFT alone (62.8). A fixed rubric was actually worse than SFT on some dimensions, suggesting generic criteria can mislead optimization.

• Two tasks beat one. Splitting diagnosis from revision improves Task 2 Overall by 0.42 points and claim–suggestion alignment by 0.35 points under identical backbone and data.

• Calibrated claim generation. ActReview-RL produces 1.24 claims per instance, close to the reference average of 1.37, whereas zero-shot baselines over-generate two or more.

• Localized retrieval is competitive but not universally better. Retrieved-chunk training matches full-paper training on aggregate benchmarks with shorter inputs, but full-paper training is better for concerns requiring cross-section reasoning (mean score 1.49 vs. 1.17).

• Generalization and abstention hold up. On 175 held-out 2025–2026 papers, ActReview-RL has the lowest severe-error rate (6.29%) among evaluated systems. It correctly abstains on 78.3% of 360 screened non-supported weakness–paper pairs while maintaining a 95.4% answer rate on supported instances.

• On incomplete rebuttals, the model does not just copy. For 50 hard cases with an incomplete rebuttal, ActReview-RL extends or rejects the original resolution 66% of the time.

Methodology in Plain English

The researchers started from real, publicly visible artifacts of peer review on OpenReview: reviewer critiques and the authors' written responses. Because a rebuttal usually explains how a concern was or could be addressed, it acts as a signal about what a useful revision looks like — but the model is never allowed to see it.

Step 1 — Extract and align. Each review is broken into atomic weakness units, one concern at a time. Each unit is then matched to the passage in the rebuttal that responds to it, using structural and lexical cues followed by semantic matching.

Step 2 — Rewrite. A language model converts each weakness–rebuttal pair into reviewer-style output: a diagnostic claim naming the paper-specific deficiency, and suggestions specifying what, where, how, and to what end. Filters remove artifacts such as leaked rebuttal content, retrospective phrasing ("we have added..."), and near-copies of the rebuttal.

Step 3 — Retrieve supporting evidence. Rather than feeding the whole paper, the system pulls localized chunks from the paper itself that ground the concern or the proposed revision. Training inputs contain only paper metadata, the weakness label, and these chunks — never rebuttal text.

Step 4 — Train in two stages. First, supervised fine-tuning on Qwen3-8B-Base teaches the output schema and reviewer register across both tasks. Then, GRPO reinforcement learning refines the model using per-instance rubrics. Each rubric is built offline for a specific paper–weakness pair by sampling varied model outputs, adding human and model-written references, and having a strong LLM synthesize hard constraints plus weighted soft requirements. Outputs that violate hard constraints get zero reward; the rest are scored by deterministic checks and semantic judging.

Step 5 — Evaluate three ways. Pairwise LLM-as-judge on 1,000 instances, pairwise human evaluation by two graduate-level annotators on a 200-instance subset (93% agreement, randomized order, hidden model identities), and complementary automatic metrics. Rebuttals are used to build data and rewards but never as evaluation input.

Why This Matters

Peer review is under strain from publication growth, and LLM tools are already being used on both sides of it. Existing tools largely stop at criticism. This work shows that the review–rebuttal record is an underexploited supervision source, and that explicitly splitting "what is wrong" from "what to do about it" produces feedback that is measurably more specific and more executable — without requiring the model to be enormous.

Real-world applications:

• Pre-submission self-review. Authors run their draft against each weakness category before submitting and get concrete revision plans rather than vague warnings.
• Reviewer assistance and training. Junior reviewers or students can see what a well-grounded, actionable critique of a given section looks like, using the same evidence-grounded scaffolding.
• Editorial and program-committee triage. Conference and journal workflows could flag submissions whose weaknesses are resolvable by revision versus those facing fundamental objections.
• Internal lab and corporate research review. Groups running internal review before external submission can generate structured, weakness-specific feedback at scale.

Industry relevance: The method targets non-verifiable generation, where no exact-match answer exists — a general problem for anyone post-training models to write open-ended expert text. The rubric-construction technique, which mines candidate failures to define per-instance reward criteria, transfers to code review, legal drafting, medical documentation, and any domain where quality is multi-dimensional and judged rather than checked. Publishers, academic-tooling companies, and LLM post-training teams all have direct stakes.

Future Directions

• Closing the technical accuracy gap. The authors identify deeper scientific judgment as the main unresolved weakness; their model makes feedback more specific and useful, not necessarily more technically correct.

• Handling non-rebuttal-resolvable concerns. The benchmark deliberately covers weaknesses that can be linked to concrete revisions, excluding novelty disputes, deep conceptual disagreements, and irreparable methodological flaws — which likely need literature-scale comparison or interactive dialogue.

• Decoupling from the judge model. GPT-5.4 is used for data enhancement, rubric construction, semantic reward scoring, and scalable pairwise evaluation. The authors mitigate this with offline rubric construction, disjoint evaluation instances, and human validation, but broader independent judging is needed.

• Reconciling localized and full-paper context. Localized retrieval wins on aggregate efficiency while full-paper input wins on cross-section reasoning. A hybrid that decides when to zoom out is an open design question.

Target Audience

NLP and LLM post-training researchers interested in reward design for open-ended tasks; meta-science and scholarly-communication researchers studying peer review at scale; builders of academic writing and review-assistance tools; and conference or journal organizers evaluating how automated feedback could fit into editorial workflows. Readers wanting only the conceptual contribution can follow the formulation and results tables without the post-training details, while those interested in rubric-based RL for non-verifiable domains will find the ablation on reward design most directly useful.

Authors’ abstract

As LLMs are increasingly used for pre-submission self-review, there is growing demand for feedback that not only identifies weaknesses but also guides authors toward concrete revisions. We study this as Actionable Peer-review Generation and decompose it into two subtasks: diagnostic claim generation and revision suggestion generation. We introduce ActReview, a rebuttal-guided post-training framework that connects paper-specific diagnoses to concrete, grounded revision plans. Our central insight is that author rebuttals reveal plausible actions for addressing reviewer concerns and can therefore provide latent supervision for revision-oriented feedback. From real review-rebuttal threads on OpenReview, we construct ActReview-40K by aligning reviewer weaknesses with author responses and grounding the resulting feedback in localized paper evidence. We post-train Qwen3-8B-Base with multi-task supervised fine-tuning followed by GRPO using candidate-aware, weakness-specific rubric rewards. We also introduce ActReview-Bench, a human-curated benchmark of 1,000 instances for evaluating diagnostic quality and revision usefulness. Experiments show that ActReview outperforms prior specialized review-generation models on actionability and grounding while remaining competitive with strong prompt-based LLMs. Human evaluation confirms improved revision usefulness while revealing a remaining gap in technical accuracy, and additional analyses support generalization to held-out papers and robustness across independent judges.