When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

AI summary

Overview

• Research area: Machine learning, specifically large language models applied to scientific peer review, recursive/synthetic-data training, and inference-time model control.
• Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, LoRA, embeddings, and activation steering, but the central argument is stated in accessible terms.
• Scope (one sentence): The paper runs a controlled one-step experiment in which an LLM reviewer is trained on mixtures of official and model-generated ICLR reviews, documents a "scientific-judgment collapse" in rating and semantic diversity, and proposes an open-source system (TrustReviewer) that combines curated training data with activation steering to mitigate it.

What This Paper Is About

As LLM-written reviews enter public review corpora, later reviewer models may be trained on the judgments of earlier ones, creating a recursive feedback loop. The authors ask a concrete version of this question: if a review model is fine-tuned on reviews generated by a previous review model, does the range of scientific judgments it expresses narrow? They build a controlled experiment to isolate synthetic-review exposure as the only changed factor, then design and evaluate a mitigation system.

Key Contributions

1. A controlled framework for recursive AI-reviewer training. The authors formalize the feedback loop and construct four successor models (M₂) that share initialization, filtering, and optimization settings, differing only in the percentage of synthetic reviews per paper (0%, 33%, 66%, 100%), where each paper contributes three reviews.
2. Evidence of scientific-judgment collapse. After one recursive training step, introducing synthetic reviews compresses the rating distribution and monotonically reduces both same-paper and corpus-level semantic diversity, even though the mean rating does not shift consistently toward leniency or harshness.
3. TrustReviewer, an open-source review system with two interventions. Training-time corpus curation reduces low-quality and semantically degenerate supervision, and test-time paired activation steering mitigates residual collapsed tendencies without further training or additional expert annotation.
4. A curated corpus and released artifacts. The curated corpus contains 112,743 paper–review examples totaling approximately 1.9 billion tokens, and the project page, dataset (Hugging Face), and code (GitHub) are released publicly.

Main Findings

• Synthetic exposure compresses rating diversity. Official reviews have a rating standard deviation of 1.73 and entropy of 2.38; the official-review-only M₂ baseline has 1.63 and 2.31. Adding 33% synthetic reviews reduces these to 1.44 and 2.14, and values remain below the baseline at 66% and 100% synthetic exposure.
• The compression is not a directional shift. Mean ratings move non-monotonically: 5.30 at 0% synthetic exposure, 5.85 at 33%, and 5.70 at 100%. The effect is homogenization of the expressed rating distribution rather than systematic leniency or harshness.
• Same-paper semantic diversity declines monotonically. Mean pairwise semantic distance falls from approximately 0.159 (0% synthetic) to 0.153, 0.147, and 0.142 at 33%, 66%, and 100%, an approximately 11% reduction from 0% to 100%.
• Corpus-level semantic diversity also contracts. Spread around the corpus centroid falls from approximately 0.609 (0%) to 0.584, 0.580, and 0.579 at 33%, 66%, and 100%, an approximately 5% reduction from 0% to 100%.
• TrustReviewer achieves the strongest recommendation agreement among evaluated models. Exact match is 75.40% ± 0.46 with MAD 1.079 ± 0.005, versus 73.10% ± 1.36 and 1.113 ± 0.022 for OpenReviewer, 61.85% ± 0.33 and 1.335 ± 0.012 for Qwen3.6-35B-A3B, and 33.93% ± 0.54 and 2.679 ± 0.010 for its Llama initialization.
• TrustReviewer improves rating entropy relative to other LLM reviewers. Entropy reaches 2.18 ± 0.01, above Llama (1.53 ± 0.03), Qwen (1.94 ± 0.02), and OpenReviewer (2.10 ± 0.05), narrowing the gap to the official reference value of 2.38. It also shows greater corpus-level semantic spread and same-paper semantic distance than those baselines in Figure 4.
• Test-time steering adds a measured gain. Relative to the same TrustReviewer checkpoint without steering, paired activation steering raises exact match from 73.85% to 75.40% (a gain of 1.55 percentage points) and rating entropy from 2.13 to 2.18, while MAD remains essentially unchanged. Same-paper semantic distance decreases slightly, and the authors state that steering does not uniformly improve every diversity metric.
• Multi-step compounding is untested. The experiment covers a single recursive step; whether scientific-judgment collapse compounds over multiple generations is left as an open question.
• The experiment is not a measurement of real AI assistance. The synthetic percentages denote only reviews explicitly generated by M₁, not the total prevalence of AI assistance in the corpus.

Methodology in Plain English

The authors start from Llama 3.1 8B (Meta-Llama-3.1-8B-Instruct) and fine-tune it on official ICLR reviews from 2018–2023 to create a first reviewer, M₁. They use M₁ to write reviews for ICLR 2024 papers, then train four successor models on ICLR 2024 data where each paper has three reviews, of which zero, one, two, or three come from M₁. The 2018–2023 period is used because those reviews largely predate widespread ChatGPT-assisted reviewing and because Llama 3.1 8B's reported training cut-off is December 2023, so ICLR 2024 reviews fall after it. All four successors share the same M₁ initialization and training configuration, making the synthetic percentage the only planned difference.

Training uses supervised fine-tuning in LlamaFactory with LoRA (rank 64, scaling factor 128, dropout 0.05, all target modules), a maximum context of 64,000 tokens, per-device batch size 1, 16 gradient-accumulation steps, warmup ratio 0.05, three epochs, bfloat16 precision, and FlashAttention-2. M₁ uses a learning rate of 2 × 10⁻⁵ with cosine decay; the M₂ successors use 2 × 10⁻⁶. A filtering pipeline removes malformed, follow-up, structurally invalid, excessively short, repetitive, and overlength reviews. For evaluation, 2,000 papers are held out (57, 88, 136, 161, 161, 235, 449, and 713 papers from 2018, 2019, 2020, 2021, 2022, 2023, 2024, and 2025 respectively) and excluded from training.

Diversity is measured in two ways. For same-paper analysis, up to three official reviews and model reviews from three independent runs are embedded and compared by average pairwise cosine distance. For corpus-level analysis, reviews of the same paper are averaged into a paper-level embedding, and the mean squared distance to the corpus centroid is computed. Embeddings use jinaai/jina-embeddings-v3 with the symmetric text-matching task at a maximum of 8,192 tokens; the 37,681 held-out official and generated reviews used all fit (maximum 5,940 tokens), so none were truncated. Generation uses temperature 0.6, top_p 0.95, repetition penalty 1.0, up to 4,096 new tokens, and three runs without fixed decoding seeds.

TrustReviewer is built differently from the controlled experiment: rather than isolating synthetic exposure, it trains one final reviewer on a curated corpus of ICLR reviews from 2018–2025, curated with the same unified filtering pipeline, in a single stage with no intermediate or recursive step. On top of that, the authors sample K = 5,000 paper–review pairs, generate a review of each paper with the trained reviewer, and estimate per-layer steering vectors as the average difference between the last-token hidden states of official reviews and generated reviews of the same papers. At inference, this vector is added at the last token position with strength α, leaving model parameters unchanged. Layer set and strength were selected using 100 separate validation pairs by exact recommendation match, choosing the final decoder layer (layer 31) and α = 0.15. Baselines are Meta-Llama-3.1-8B-Instruct, OpenReviewer, and Qwen3.6-35B-A3B.

Why This Matters

The paper identifies a specific risk in how AI-assisted peer review could feed back into AI training: if later reviewer models learn from earlier models' judgments, the diversity of scientific judgment can shrink even when average scores look normal. Because the effect appears in rating spread and semantic diversity rather than in a shift toward kinder or harsher scores, it would not be visible from average-score tracking alone. The authors also release the curated corpus, which they note OpenReviewer does not do, supporting reproducible study of reviewer training.

Real-world applications:

• Conference and journal review pipelines: provides measurements and a released system for evaluating whether AI-assisted or batch-generated reviews are narrowing the distribution of judgments in a venue's review corpus.
• Training-data curation for domain models: the filtering criteria (removing follow-up markers, very short reviews, duplicated lines, repeated 30-grams, and structurally invalid reviews) are directly reusable for building review or critique datasets.
• Inference-time behavior control: paired activation steering shows a way to adjust a deployed model's output characteristics without retraining or collecting new expert labels.
• Benchmarking LLM reviewers: the exact-match, MAD, and entropy metrics, following Idahl and Ahmadi (2025), give a reproducible evaluation protocol for comparing review-generating models.

Industry relevance: organizations that train or deploy LLMs for document review, evaluation, or critique — including publishers, grant agencies, and platform companies building evaluation tooling — face the same recursive-data question. The results also matter for any workflow where model outputs become future training data, since the paper shows a measurable diversity loss from a single recursive step at a 33% synthetic fraction.

Future Directions

1. Multi-generation recursion. The study covers one recursive step only; the authors state that whether scientific-judgment collapse compounds over multiple generations remains an important question.
2. Separating the sources of collapse. Because the experiment varies only the explicit synthetic fraction, it does not quantify how much unobserved AI assistance is already present in "official" reviews or how that affects later training.
3. Extending and validating the steering intervention. Steering improves exact match and entropy but slightly reduces same-paper semantic distance, so the authors note it does not uniformly improve every diversity metric; the selection was made on 100 validation pairs at layer 31 with α = 0.15.
4. Limitations are deferred. The paper states that the study's limitations are discussed in Appendix G, whose contents are not included in the provided text; the acknowledgment notes support from DARPA (HR001124S0029-AIQ-FP-019), the NSF TRAILS Institute (2229885), Open Philanthropy, Apple, and the NAIRR Pilot.

Target Audience

This paper is most useful to researchers working on LLM-based scientific review and evaluation, to machine learning practitioners studying recursive or synthetic-data training and model collapse, and to conference organizers, editors, and program chairs who need to understand how model-generated reviews may affect future review quality. It is also relevant to engineers building evaluation or critique systems where model outputs feed back into later training data. Readers should be comfortable with fine-tuning terminology, though the core findings — compressed rating distributions and reduced semantic diversity — are stated plainly enough for a policy-oriented audience.

Authors’ abstract

Large language models (LLMs) increasingly participate in scientific evaluation, both as automated reviewers and as assistants to human reviewers. As model-generated reviews enter public data and future training corpora, AI peer review can become recursive: later reviewers learn from judgments produced by earlier models. We study one step of this feedback loop in a controlled setting. Starting from Llama 3.1 8B, we first fine-tune a reviewer on official ICLR reviews from 2018--2023 and then train four successor models on ICLR 2024 data with systematically varied mixtures of official and model-generated reviews. Our study shows that introducing synthetic reviews compresses rating distributions and reduces both same-paper and corpus-level semantic diversity. We call this pattern $\textbf{scientific-judgment collapse}$. To mitigate this failure mode, we introduce $\textbf{TrustReviewer}$, an open-source LLM-based system for generating peer reviews of AI and machine learning papers. TrustReviewer intervenes at two complementary stages. For training-time prevention, we train the core reviewer in a single stage on a curated corpus designed to reduce low-quality and semantically degenerate supervision. For test-time correction, paired activation steering aims to further mitigate residual tendencies toward collapsed judgments without further training or additional expert annotation. Together, these results characterize a concrete risk of recursive reviewer training and provide practical interventions for preserving judgment diversity and improving recommendation alignment in AI-assisted scientific evaluation.