Automatic Paper Reviewing with Heterogeneous Graph Reasoning over LLM-Simulated Reviewer-Author Debates

AI summary

Overview

Research area: Automated peer review (natural language processing), combining large language model (LLM) multi-agent simulation with heterogeneous graph neural networks.

Technical level: Intermediate. Readers need basic familiarity with LLMs and graph neural networks, but the paper explains its graph construction and reasoning steps in accessible terms.

Scope: The paper introduces ReViewGraph, a framework that simulates multi-round reviewer–author debates with LLM agents, encodes the resulting opinions and their relations as a typed heterogeneous graph, and reasons over that graph to predict an accept/reject decision.

What This Paper Is About

Automatic peer review systems either prompt an LLM directly, which produces shallow, hallucination-prone and prompt-sensitive reviews, or fine-tune open-source LLMs on review data, which suffers from scarce and biased data and yields only single-perspective opinions. Neither style captures the argumentative back-and-forth of real peer review, where reviewers disagree with each other and authors rebut critiques. ReViewGraph's goal is to explicitly model that debate structure as a heterogeneous graph and reason over it to reach a more informed, interpretable review decision.

Key Contributions

1. A debate-based reviewing framework. The authors propose ReViewGraph, which models reviewer–author interactions as heterogeneous graphs built from LLM-simulated multi-round debates rather than from raw manuscript text alone.
2. A structured heterogeneous debate graph. The graph uses semantically typed nodes and edges to capture fine-grained argumentative relations across multiple review perspectives, and a graph neural network performs relational reasoning over this structure.
3. Strong empirical results without fine-tuning. Experiments on three datasets (ICLR 2023, 2024, 2025) show ReViewGraph outperforming 7 strong baselines, with an average relative improvement of 15.73% over the second-best models, while requiring no LLM parameter updates.
4. Component-level validation. Ablation studies and two case studies show which graph elements matter most (particularly title nodes and evaluation-dimension nodes) and demonstrate correct handling of subtle negative consensus and isolated dissent.

Main Findings

• Best on all three datasets and all four metrics. ReViewGraph scores on ICLR 2023: Accuracy 70.29, Macro Precision 69.85, Macro Recall 69.92, Macro F1 69.89. On ICLR 2024: 66.10, 65.48, 65.73, 65.54. On ICLR 2025: 71.65, 72.04, 72.01, 71.65, which the paper calls its strongest result, with all metrics exceeding 70.
• Average relative improvement of 15.73%. The abstract and introduction report this figure as the average relative improvement over the second-best baselines. The conclusion instead phrases it as a relative improvement of 15.73% in Macro F1 over the best-performing baseline.
• Statistically significant gains. Two-sample T-tests against the best-performing baseline, CycleReviewer-70B, gave p-values of 0.0192 for accuracy and 0.0067 for F1, both below the 0.05 threshold.
• Large margin over the strongest fine-tuned baseline on ICLR 2025. ReViewGraph's Macro F1 of 71.65 exceeds CycleReviewer-70B by more than 12 percentage points on that dataset.
• Prompt-based baselines lag by a wide margin. Compared with the ICL-based and CoT-based methods, ReViewGraph improves by more than 10 percentage points on average in Accuracy and Macro-F1.
• GraphEval, the graph-based baseline, performs weakest overall. Its accuracy is 50.00 on ICLR 2023, 48.97 on ICLR 2024, and 46.91 on ICLR 2025; the paper attributes this to its reliance on sentence-level similarity within abstracts rather than explicit labeled relations across full-text interactions.
• Title nodes are the most important graph component. In ablations, removing paper title nodes (w/o Title) causes the largest drop: accuracy falls to 67.03 on ICLR 2023, 64.73 on ICLR 2024, and 66.49 on ICLR 2025, versus 70.29, 66.10 and 71.65 for the full model.
• Other components contribute progressively. Removing evaluation dimension nodes (w/o Eval), reviewer–author interaction edges (w/o RAR), or inter-reviewer relation edges (w/o IRR) each produces moderate declines, and collapsing the graph into a homogeneous structure (w/o Hetero) also impairs performance.
• Case studies show nuanced reading of reviewer dynamics. In a rejection case, ReViewGraph detected a negative consensus across theoretical rigor and baseline comparison despite polite language, while ICL, CoT prompting and LLM dialogue agents predicted Accept. In an acceptance case, it correctly treated Reviewer 2's suggestion as isolated and non-decisive where baselines predicted Reject.
• No fine-tuning required. The paper emphasizes that ReViewGraph updates no LLM parameters, unlike CycleReviewer and DeepReview, which the authors say gives it greater generalizability, training efficiency and inference controllability.

Methodology in Plain English

The pipeline has three parts.

Simulated debate. Each paper is processed by four role-specific agents plus a coordinator: three regular reviewer agents, one author agent, and one senior reviewer agent acting as a meta-level coordinator. All agents use Qwen2.5-VL-72B-Instruct as the backbone because it handles both text and images. The debate runs in three stages: initial review (each reviewer lists strengths, concerns and weaknesses), author rebuttal (the author produces a point-by-point response), and re-evaluation (reviewers revise or reaffirm their positions in light of the rebuttal).

Graph construction. From the debate transcript, the framework builds a heterogeneous graph with four node types — Title, Evaluation Dimension, Reviewer Opinion and Author Opinion — and four meta-relation types. Paper-to-dimension edges link a paper to four evaluation dimensions: Methodological Novelty, Experimental Completeness, Motivation Clarity and Writing Fluency. Dimension–opinion edges attach each reviewer opinion to a dimension. Inter-reviewer edges are labeled agree, disagree, complement, progressive or independent. Reviewer–author edges are labeled accept, reject, clarify, compromise, extend or neutral. GPT-4.1-mini is prompted to extract opinion triplets and to classify each opinion into a dimension.

Reasoning and prediction. A Heterogeneous Graph Transformer (HGT) applies heterogeneous mutual attention, heterogeneous message passing and target-specific aggregation, using relation-specific projection weights. Node and edge representations are initialized with Qwen3-Embedding-8B. Node embeddings are mean-pooled per node type, the pooled vectors are concatenated, and a two-layer feedforward network with softmax predicts accept or reject. The appendix reports a hidden dimension of 128, up to 100 training epochs, batch size 32, the Adam optimizer with a learning rate of 1×10⁻⁴, and early stopping on validation F1. The appendix describes the graph Transformer as a two-layer HGT "with a total of three layers." Papers from OpenReview (ICLR 2023, 2024, 2025) were processed with the MinerU tool to extract text, figures and tables; appendices were excluded. The dataset counts total 951 training, 134 validation and 276 test papers for 2023; 1012, 143 and 292 for 2024; and 1343, 191 and 388 for 2025.

Why This Matters

The paper targets a concrete bottleneck: submission volumes are straining peer review, and LLM reviewers have been trialed at venues such as ICLR 2025 and announced for AAAI 2026. ReViewGraph's central claim is that representing who said what, in response to whom, and under which evaluation criterion is more reliable than asking a model for a score. It also matters because the method is training-free with respect to the LLM, making it cheaper to deploy than fine-tuned reviewer models while producing an inspectable graph.

Real-world applications:

• Assisting program committees at conferences by producing an additional structured opinion alongside human reviews.
• Flagging consensus and dissent patterns among reviewers to help area chairs and meta-reviewers identify decisions that need closer scrutiny.
• Auditing review quality by exposing which evaluation dimensions received little or contradictory coverage.
• Journals and workshops with limited reviewer pools needing a first-pass triage signal over large submission batches.

Industry relevance: Any organization that runs a submission-and-review pipeline — academic publishers, conference organizers, industrial research labs with internal review processes, and grant programs — could use structured debate graphs as an interpretable decision aid. The method also has broader relevance for building multi-agent LLM systems whose outputs must be auditable rather than a single opaque score.

Future Directions

• Extending beyond accept/reject. The paper predicts only a binary decision; generating full review text or calibrated scores, as ReviewRobot, CycleReviewer and DeepReview attempt, is not addressed.
• Reducing dependence on LLM extraction. Graph instantiation relies on in-context prompting with GPT-4.1-mini to extract triplets and dimensions, so hallucinated or missing relations remain a possible failure source the paper does not quantify.
• Testing generalization beyond ICLR. All three datasets come from OpenReview ICLR 2023, 2024 and 2025 in computer science; behavior on other venues, disciplines or review formats is untested here.
• Richer agent designs. The framework uses three regular reviewers, one author and one senior reviewer with a fixed set of four evaluation dimensions; whether more varied reviewer personas or different dimension sets improve results is left open.

Target Audience

Researchers and practitioners working on LLM-based peer review, multi-agent LLM systems, and graph neural networks applied to text. It is also relevant to program chairs, meta-reviewers and research-integrity groups evaluating how automated reviewing tools perform, and to industry teams building auditable multi-agent reasoning pipelines. Readers wanting implementation details should consult the code at https://github.com/relic-yuexi/ReViewGraph, which the paper states contains the ReViewGraph and baseline implementation details, with code and data also provided in the supplementary materials.

Authors’ abstract

Existing paper review methods often rely on superficial manuscript features or directly on large language models (LLMs), which are prone to hallucinations, biased scoring, and limited reasoning capabilities. Moreover, these methods often fail to capture the complex argumentative reasoning and negotiation dynamics inherent in reviewer-author interactions. To address these limitations, we propose ReViewGraph (Reviewer-Author Debates Graph Reasoner), a novel framework that performs heterogeneous graph reasoning over LLM-simulated multi-round reviewer-author debates. In our approach, reviewer-author exchanges are simulated through LLM-based multi-agent collaboration. Diverse opinion relations (e.g., acceptance, rejection, clarification, and compromise) are then explicitly extracted and encoded as typed edges within a heterogeneous interaction graph. By applying graph neural networks to reason over these structured debate graphs, ReViewGraph captures fine-grained argumentative dynamics and enables more informed review decisions. Extensive experiments on three datasets demonstrate that ReViewGraph outperforms strong baselines with an average relative improvement of 15.73%, underscoring the value of modeling detailed reviewer-author debate structures.