InternReviewer & InternAdvocate: Objective Reward and Evaluation for Agentic Reinforcement Learning in Peer Review and Rebuttal

AI summary

Overview

Research area: Automated scientific peer review and author rebuttal generation using LLM-based agents, with a focus on reinforcement learning reward design and evaluation methodology.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (policy optimization, group-relative advantages, clipping objectives), Mixture-of-Experts architectures, retrieval-augmented generation, and dense retrieval infrastructure.

Scope: The paper introduces two trained scholarly agents, InternReviewer (peer review) and InternAdvocate (rebuttal), together with a large-scale OpenReview-derived dataset, a local arXiv retrieval index, a multi-component objective reward function, and a multi-dimensional evaluation protocol.

What This Paper Is About

Automated peer review with LLMs is unreliable because models hallucinate technical flaws and cite papers that do not exist, while existing training approaches either rely on subjective "LLM-as-a-judge" reward signals or on supervised fine-tuning that only imitates the surface style of human reviewers. This paper builds specialized agents for review and rebuttal that retrieve real arXiv literature during generation and are trained with a rule-based, multi-dimensional reward that verifies every citation against actual tool-use logs. The goal is to replace subjective judging with an objective, verifiable training and evaluation signal for scholarly writing.

Key Contributions

1. A large-scale scholarly interaction dataset. The authors curate 184,857 unique review-rebuttal pairs with their original manuscript PDFs from OpenReview, covering ICLR (2013–2025), NeurIPS (2021–2024), and ICML (2025). Final ratings are deliberately excluded because terminal post-rebuttal scores are often decoupled from the initial critique. Papers are parsed with MinerU into structured Markdown, truncated to the main body without References, and anonymized for double-blind review.

2. InternReviewer and InternAdvocate, trained by Agentic RL without SFT cold start. Because real scholarly data lacks intermediate tool-use trajectories, the authors skip supervised fine-tuning and train directly with reinforcement learning. Both agents invoke an arxiv_search tool at least once per session and wrap their output in <reviewer> or <rebuttal> tags so that the scholarly output can be separated from the raw trajectory for noise-free reward computation.

3. An objective, decomposed reward function. The total reward is a linear sum of four modular terms — r_polar + r_format + r_tool + r_cite — covering reference-anchored semantic alignment, structural compliance, tool-use behavior, and citation grounding. Hallucination is suppressed by a zero-tolerance penalty of −1.0 if any citation in the final output cannot be matched against the retrieval logs.

4. A high-throughput time-aware retrieval infrastructure and evaluation protocol. A local Elasticsearch cluster indexes 593,092 computer science manuscripts from arXiv, with hierarchical summaries and dense vectors, plus asymmetric temporal cutoffs that differ for reviewer and advocate roles. The evaluation protocol moves beyond BLEU and paper-score prediction to combine semantic metrics (POLAR-7B, Qwen3-Reranker-8B) with agentic format metrics.

Main Findings

• Agents improve under the closed-loop framework: The abstract reports that agents trained within this framework exhibit significant improvements in reasoning depth and citation accuracy. Specific numeric improvements are not reported in the available content.

• Quantitative results are not reported in the available text: The provided paper content is truncated within Section 4.2 (Evaluation Metrics), so the main results tables, benchmark scores, and ablation numbers are not present. Any specific performance figures would need to be read from the full paper.

• Subjective reward signals are unstable: The authors argue that "LLM-as-a-judge" rewards and preference pairs built on static rubrics produce inconsistent signals across models and prompting strategies, causing training instability and pushing models toward stylistic mimicry over factual accuracy.

• Lexical overlap metrics are inadequate for this domain: BLEU-1–4 and ROUGE-1/2/L reduce quality to surface token matching. The paper notes that a review using "inadequate" instead of "insufficient" would receive no credit despite identical meaning, while a review mirroring the reference vocabulary but arguing the opposite conclusion would be rewarded. These metrics are therefore treated as secondary diagnostics rather than primary selection criteria.

• Prior training-based systems lack tool use: DeepReviewer fine-tunes a 7B model on approximately 13,000 ICLR papers, and ReviewRL applies RL after SFT; both train without tool use, limiting their ability to assess novelty against the actual research landscape. The authors also note these training sets are small and temporally concentrated, raising generalization questions.

• Training-free systems depend on proprietary models: Systems such as CSPaper, the AAAI AI Review system, Stanford PaperReview, OmniScientist, MAMORX, REMOR, and ReviewerToo offer no pathway to domain-specific improvement and tend toward formulaic "sycophantic" reviews.

• Sequence-level optimization was chosen for stability: GSPO is used instead of token-level GRPO because token-level updates in MoE architectures can cause drastic shifts in expert routing distributions and ultimately model collapse.

Methodology in Plain English

The task is framed as a sequential decision process. The agent sees the manuscript, the dialogue history, previously retrieved evidence, and its own prior reasoning. At each step it either issues a search query or writes text. The goal is to maximize cumulative reward over the trajectory.

Data. Papers and their review-rebuttal threads are pulled from OpenReview. PDFs are converted to structured Markdown with MinerU so that section hierarchies survive. The text is cut to the main body — References removed — because conference guidelines do not oblige reviewers to read appendices, and the main body is typically limited to 8–9 pages. Author names, affiliations, and acknowledgments are stripped for double-blind simulation. InternReviewer sees only the anonymized manuscript; InternAdvocate sees the manuscript plus a first-round critique.

Retrieval. Instead of calling an external API, the authors run a local Elasticsearch cluster over 593,092 arXiv computer science manuscripts. Each paper is split into segments of 8,000 tokens with 800 tokens of overlap, summarized by Qwen3-235B-A22B-Thinking-2507 into a "Global Snapshot," and embedded with Qwen3-Embedding-8B. Caching the summaries makes retrieval fast, deterministic, and reproducible — the same paper always yields the same information to every agent.

Temporal guardrails. InternReviewer can only retrieve papers published at least 120 days before the submission deadline, matching what a human reviewer could realistically have read. InternAdvocate's cutoff is relaxed to 90 days after the deadline, reflecting that authors may cite concurrent work during rebuttal. This asymmetry prevents future-knowledge leakage while mimicking the real division of information between reviewers and authors.

Reward. Four components. The semantic term uses POLAR-7B to compare the generated text against the human reference, with the References section excluded from both, then normalizes the raw score through 2 · tanh(s_POLAR / 10) because POLAR's raw scores have mean 0 and standard deviation 10. The format term rewards presence of required sections and a character count between 2,000 and 10,000; InternReviewer must produce Summary, Strengths, Weaknesses, Questions, and References, while InternAdvocate must produce Responses and References. The tool term gives −1.0 for no tool calls, +1.0 for 4–6 interaction steps, and −0.6 otherwise, discouraging both laziness and infinite retrieval loops. The citation term checks that inline citations match the reference list (+1.0 or −1.0), deducts −0.6 for duplicated entries and −0.3 for non-sequential numbering, penalizes citation-to-source ratios above 3.5, and applies −1.0 for any fabricated reference.

Optimization. GSPO computes the policy update ratio over whole sequences rather than individual tokens, with length normalization. The advantage is the group-normalized reward. This keeps MoE expert routing stable while allowing multi-step reasoning and tool use.

Evaluation. Models are compared against Claude Sonnet 4.5, Gemini 3.1 Pro Preview, GPT-5.2, and Qwen3-30B-A3B-Thinking-2507, all run through the same agentic evaluation framework with the same search tool. Training uses 72K samples for one epoch with a 1K held-out test set. Metrics include BLEU-1–4, ROUGE-1/2/L, POLAR-7B, Qwen3-Reranker-8B, and the format score.

Why This Matters

Peer review is under severe strain from rising submission volumes, producing reviewer fatigue and inconsistent feedback, and naive LLM assistance risks fabricating flaws or citing nonexistent literature. This work reframes the problem around verifiability: citations are checked against actual retrieval logs, so the training signal itself penalizes hallucination rather than merely discouraging it in a prompt.

Real-world applications:

• Assisting conference reviewers by producing literature-grounded first-pass critiques that can be audited against a real retrieval record.
• Helping authors draft structured rebuttals that cite verifiable concurrent work published during the review period.
• Providing editorial and program-committee triage with a measurable structural and citation-integrity signal rather than opaque model confidence.
• Serving as a reproducible evaluation suite for any future scholarly agent, since the retrieval cache makes scoring deterministic across runs and models.

Industry relevance: The framework generalizes to any long-form, evidence-grounded expert writing task where claims must trace back to retrieved sources — legal analysis, technical due diligence, clinical literature review, and regulatory drafting. The infrastructure lessons also matter directly: replacing rate-limited external APIs with a local index and precomputed summaries removes the I/O bottleneck that would otherwise stall large-scale RL rollouts, and the sequence-level optimization choice is a practical recipe for training MoE models without routing collapse.

Future Directions

• Report and analyze the full experimental results. The available text cuts off before the results tables, so the actual magnitude of improvement over Claude Sonnet 4.5, Gemini 3.1 Pro Preview, GPT-5.2, and the Qwen3-30B-A3B-Thinking-2507 starting point is not yet established here.

• Generalize beyond the curated venues. Training and evaluation draw on ICLR, NeurIPS, and ICML. Whether the same reward design transfers to journals, non-CS disciplines, or venues with different review cultures is untested.

• Retire the human review as the gold standard. The semantic reward treats human reviews as the benchmark, which the authors acknowledge inherits the biases and noise of human reviewers — the very problem SFT was criticized for.

• Validate the reward weights. Parameters such as the 120-day and 90-day retrieval cutoffs, the 4–6 step tool-use window, the 2,000–10,000 character length band, the 3.5 citation-to-source ratio, and the −1.0 hallucination penalty are specified but their sensitivity is not examined in the available content.

• Address the dual-use risk. Deployed at scale, such agents could shift reviewing effort rather than reduce it, or be used by authors to mass-produce rebuttals, and reviewing integrity mechanisms are not discussed.

Target Audience

Researchers working on LLM agents, reinforcement learning for long-form generation, and retrieval-augmented systems; program committee members and journal editors interested in AI-assisted reviewing; and practitioners building reward models or evaluation pipelines for subjective, open-ended tasks where verifiability of claims is the central engineering constraint.

Authors’ abstract

Generating professional scholarly content, such as peer reviews and rebuttals, requires an intricate synergy between domain reasoning and factual grounding. This work presents a comprehensive framework for the development and evaluation of specialized scholarly agents, InternReviewer and InternAdvocate. We first establish a large-scale, high-quality scholarly dataset and integrate a high-efficiency arXiv retrieval tool to enable active evidence gathering. To optimize these agents, we implement an agentic Reinforcement Learning (RL) paradigm driven by a unified objective metric and reward system. This system avoids the biases of subjective model-based judging by employing multi-dimensional criteria, including reference-anchored semantic alignment, structural compliance, and a strict verification mechanism that cross-checks citations against real-time interaction logs to eliminate hallucinations. Experimental results demonstrate that agents trained within this closed-loop framework exhibit significant improvements in reasoning depth and citation accuracy.