RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mind

AI summary

Overview

Research area: Natural language processing, specifically LLM agents and machine Theory of Mind, applied to academic peer review and author rebuttal writing.

Technical level: Advanced. The work combines supervised fine-tuning, reinforcement learning (GRPO), reward-model construction, and automated/human evaluation pipelines.

Scope: The paper introduces RebuttalAgent, a framework that treats academic rebuttal as strategic persuasion under information asymmetry rather than surface-level linguistic imitation, and supports it with a synthetic dataset (RebuttalBench) and a specialized evaluator (Rebuttal-RM).

Note on title: the paper listing gives the title as "RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mind," while the paper content header reads "Dancing in Chains: Strategic Persuasion in Academic Rebuttal via Theory of Mind." Authors are Zhitao He, Zongwei Lyu, and Yi R. (May) Fung of the Hong Kong University of Science and Technology (arXiv:2601.15715v3 [cs.CL], listed as published 2026-01-22, content dated 25 Feb 2026).

What This Paper Is About

Academic rebuttal is framed as a dynamic game of incomplete information: authors must persuade reviewers while being unaware of reviewers' knowledge base, intrinsic biases, or how their responses will cascade. Existing approaches rely mainly on supervised fine-tuning over review datasets, which produces superficially polite but formulaic text because it imitates surface-level language rather than reasoning about the reviewer's perspective. The goal is to build an agent that explicitly models the reviewer's mental state, plans a persuasion strategy, and then writes an evidence-grounded response.

Key Contributions

1. RebuttalAgent and the TSR framework. The first framework to ground academic rebuttal in Theory of Mind, operationalized as a ToM-Strategy-Response pipeline: model the reviewer's mental state hierarchically, formulate a persuasion strategy, then generate an evidence-based response.

2. RebuttalBench. A large-scale synthetic dataset of over 70K high-quality samples built with a critique-and-refine pipeline and a mixture of teacher models (for example GPT-4.1 and Claude 3.5), where each sample contains a complete ToM-strategy-response chain. Its source is the Re²-rebuttal dataset.

3. Two-stage training with a self-reward mechanism. Supervised fine-tuning on Qwen3-8B instills TSR reasoning, followed by reinforcement learning with Group Reward Policy Optimization (GRPO) using a self-reward signal, so no separate externally annotated reward model is needed during training.

4. Rebuttal-RM. A specialized evaluator built on Qwen3-8B and trained on over 100K multi-source samples, reported to align more closely with human preferences than GPT-4.1.

Main Findings

• Automated benchmark gains. RebuttalAgent achieves the highest overall average score of 9.42 on R2-test, above o3 (9.21) and GPT-4.1 (8.50). Against the Qwen3-8B baseline (7.96), it improves by an average of 18.3%, with the largest gains in Persuasiveness and Constructiveness (up to 34.6%). It records Clarity of 9.43 and Persuasiveness of 9.20.

• Evaluator alignment. Rebuttal-RM reaches the highest average alignment score of 0.812, surpassing GPT-4.1 and DeepSeek-r1 by 9.0% and 15.2% respectively. Its per-dimension scores include Attitude 0.839, Clarity 0.821 (average of the r, β, and f metrics), Persuasiveness 0.820 (with r = 0.839), and Constructiveness 0.810.

• Human evaluation. On 100 randomly sampled comments (balanced between in-domain and out-of-domain), scored blindly by three annotators with at least three years of AI/ML research experience and top-tier reviewing experience, RebuttalAgent attains the highest average of 9.57, ahead of o3 (9.26) and GPT-4.1 (8.99). Its Persuasiveness score is 9.34, a 7.36% improvement over GPT-4.1. Inter-annotator agreement is Cohen's κ = 0.79. The human-evaluation table also lists a variant using a GPT-4.1 reward at an average of 9.59.

• Ablations confirm each component. Removing ToM (9.04), Strategy (9.31), or Thinking (9.37) reduces performance. Training ablations: SFT-only 8.27, RL-only 8.79, DPO variant 8.68. Among reward signals, response quality is the most impactful: removing R_Response drops performance to 8.63. Alternative reward sources score lower than the self-reward (RebuttalRM-reward 9.45, GPT-4.1-reward 9.35, Llama-based 9.20, Qwen3-4B-based 8.98).

• Model-agnostic effects. Applying the framework to smaller backbones raises Llama-3.1-8B from 7.44 to 9.20 and Qwen3-4B from 7.69 to 8.98.

• ToM reasoning transfers. Supplying the model-generated ToM analysis and Strategy as context to Qwen3-8B and Llama-3.1-8B yields consistent gains across categories; Qwen3-8B gains 21.0% in Presentation when guided by the full ToM and Strategy.

• Comment extraction reliability. Manual verification on 100 randomly sampled reviews finds 98% accuracy in extracting discrete comments.

• Evaluation data. R2-test contains 6,000 comments from the Re² dataset, sourced from 24 conferences and 21 workshops on OpenReview (2017–2023). The out-of-domain Rebuttal-test is built from over one thousand recent post-2023 ICLR and NeurIPS reviews, yielding 2K comments.

• Baseline comparison for human-written rebuttals. RebuttalFT, a Qwen3-8B model fine-tuned directly on real human-written rebuttals from the Re²-rebuttal dataset, scores 6.35 on the automated benchmark and 6.75 in human evaluation.

Methodology in Plain English

1. Break reviews into single comments. An LLM-as-Extractor prompt separates raw review text into discrete, unedited critical statements, so the system addresses one point at a time rather than a whole noisy review.

2. Pull only relevant manuscript context. The paper is split into paragraph-level chunks, encoded with Qwen3-Embedding-0.6B alongside each comment, ranked by cosine similarity, and the top-k chunks are retrieved.

3. Reason about the reviewer before writing. The TSR framework first builds a hierarchical reviewer profile. The macro level infers Overall Stance, Overall Attitude, Dominant Concern, and Reviewer Expertise to set global tone and strategy. The micro level classifies each comment by Significance, Methodology, Experimental Rigor, and Presentation. A strategy step then converts this profile into an explicit plan, forcing the model to decide how to respond before deciding what to write.

4. Generate the response. The final response combines strategic inputs, retrieved evidence chunks, and, during data synthesis only, the original human response as a phrasing and structure reference. The original response is not used at inference time.

5. Build training data. GPT-4.1 parses the Re²-rebuttal corpus into over 200K comment-response pairs. Comments requiring new, unprovided experiments are deliberately excluded to prevent fabrication. A curated subset of 70K comments is used: 60K filtered by category plus 10K selected at random. Teacher models then synthesize analysis, strategy, and response, concatenated with explicit <Analysis>, <Strategy>, and <Response> tags.

6. Train in two stages. Supervised fine-tuning on Qwen3-8B teaches the TSR structure. Reinforcement learning with GRPO then optimizes a self-reward that is a weighted sum of four signals: binary Format Adherence, Reasoning Quality (model-judged), Response Quality (model-judged on persuasiveness, clarity, and evidence use), and Response Diversity (scored by distance from pre-defined templated negative samples).

7. Evaluate with a purpose-built judge. Rebuttal-RM is trained on over 102K instances drawn from three sources: 12,000 original author responses as a human baseline, GPT-4.1-refined responses as a quality ceiling, and diverse model-generated replies (for example from Qwen2.5-3B and Claude 3.5) for style coverage. Labels come from a hybrid strategy: manual scoring for author responses where the reviewer subsequently raised their score, and Gemini 2.5 Pro automation for the model-generated replies. A 90% train / 10% test split is used. Evaluation metrics include Mean Absolute Error, Pearson, Spearman, and Kendall correlations, plus coarse-grained accuracy over four tiers (scores 1–3 Unconvincing, 4–6 Acceptable, 7–8 Good, 9–10 Excellent) and fine-grained accuracy over seven narrower ranges.

Why This Matters

Peer review rebuttal is a high-stakes, poorly supported stage of the research workflow. Many strong papers are weakened not by technical flaws but by unclear, unfocused, or non-strategic responses under severe information asymmetry. This paper reframes the problem as strategic reasoning rather than politeness mimicry, and supplies both a training corpus and an evaluator that others can reuse. The authors include an explicit disclaimer that generated rebuttal content is for reference only to inspire authors and assist in drafting, and is not intended to replace the author's own critical analysis and response.

Real-world applications:

• Drafting assistance for authors facing tight rebuttal windows, helping prioritize which critiques to contest directly and which to reframe.
• Support for non-native English speakers and early-career researchers who may be disadvantaged by communication style rather than scientific substance.
• Automated triage and quality scoring of rebuttals, using Rebuttal-RM as a cheaper, locally hosted alternative to proprietary judges such as GPT-4.1.
• Training and meta-science analysis, including studying what kinds of rebuttal arguments are associated with reviewers subsequently raising their scores, which is the labeling signal used for human responses in Rebuttal-RM training.

Industry relevance: academic publishers, peer-review platforms, preprint and conference management systems, and providers of scholarly writing tools could integrate a specialized rebuttal agent and evaluator. The self-reward and reward-model design is also directly transferable to other persuasion or negotiation-style generation tasks where no expert-annotated reward exists.

Future Directions

• Handling critiques that require new experiments. The dataset deliberately excludes comments needing unprovided experiments. Extending the agent to coordinate with experimental planners, or to phrase honest deferrals constructively, remains open.
• Wider domain generalization. Evaluation covers Re²-derived comments (24 conferences and 21 workshops, 2017–2023) and post-2023 ICLR and NeurIPS reviews. Whether TSR transfers to other fields, languages, or non-OpenReview review formats is not reported.
• Tightening ToM integration. Supplying generated ToM analyses as external context already improves base models, but the full agent still outperforms that setup, suggesting room for more deeply internalized perspective-taking rather than prompt-supplied profiles.
• Reward robustness and evaluation scale. The paper discusses vulnerability to reward hacking (particularly for the diversity reward) in an appendix, and the human evaluation uses 100 comments with three annotators. Larger-scale human validation and closer study of reward-gameability are natural next steps. The provided content does not include an explicit future work section from the authors.

Target Audience

NLP and LLM-agent researchers working on reasoning, persuasion, reward modeling, or reinforcement learning; authors and reviewers who want to understand what makes a rebuttal persuasive; developers building peer-review or scholarly writing tools; and meta-science researchers studying evaluation and reviewer behavior. Readers should be comfortable with supervised fine-tuning, RL optimization, and reward-model evaluation to get the most from the methodology and results sections.

Authors’ abstract

Although artificial intelligence (AI) has become deeply integrated into various stages of the research workflow and achieved remarkable advancements, academic rebuttal remains a significant and underexplored challenge. This is because rebuttal is a complex process of strategic communication under severe information asymmetry rather than a simple technical debate. Consequently, current approaches struggle as they largely imitate surface-level linguistics, missing the essential element of perspective-taking required for effective persuasion. In this paper, we introduce RebuttalAgent, the first framework to ground academic rebuttal in Theory of Mind (ToM), operationalized through a ToM-Strategy-Response (TSR) framework that models reviewer mental state, formulates persuasion strategy, and generates evidence-based response. To train our agent, we construct RebuttalBench, a large-scale dataset synthesized via a novel critique-and-refine approach. Our training process consists of two stages, beginning with a supervised fine-tuning phase to equip the agent with ToM-based analysis and strategic planning capabilities, followed by a reinforcement learning phase leveraging the self-reward mechanism for scalable self-improvement. For reliable and efficient automated evaluation, we further develop Rebuttal-RM, a specialized evaluator trained on over 100K samples of multi-source rebuttal data, which achieves scoring consistency with human preferences surpassing powerful judge GPT-4.1. Extensive experiments show RebuttalAgent significantly outperforms the base model by an average of 18.3% on automated metrics, while also outperforming advanced proprietary models across both automated and human evaluations.