Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models

AI summary

Overview

Research area: LLM evaluation integrity and membership inference — specifically data contamination detection during Reinforcement Learning (RL) post-training.

Technical level: Intermediate. Readers will benefit from familiarity with LLM training stages (pre-training, SFT, RL post-training), token-level entropy, and basic membership inference metrics such as AUC.

Scope: This paper introduces the first detection method and benchmark for identifying whether specific samples leaked into an LLM's RL post-training data, exploiting the entropy signature of RL-induced policy collapse.

What This Paper Is About

Benchmarks used to evaluate LLMs can silently end up in the models' training data, inflating reported scores. Prior detection methods target pre-training and Supervised Fine-Tuning, where models learn by mimicking ground-truth text — leaving a clear statistical trace. This paper shows that RL post-training breaks those methods (they perform near random guess) and proposes a new detector, Self-Critique, that works because RL training forces models to memorize favored reasoning paths, which is observable through entropy patterns.

Key Contributions

1. First systematic study of RL-phase contamination detection. The paper formalizes contamination in RL post-training as a black-box membership inference problem and explains why the reward-driven objective invalidates likelihood-based detectors.
2. Self-Critique, an entropy-based detector. The method prompts the model for an initial answer, then asks it to critique and revise that answer, comparing the token-level entropy sequences of the two responses. High similarity indicates policy collapse and memorization.
3. RL-MIA benchmark. A new controlled benchmark that simulates RL-phase contamination by injecting a known subset of evaluation items into RL training corpora across math (AIME 2024/2025) and synthetic logic (Knights & Knaves, SAT) tasks.
4. Strong empirical results across models and RL algorithms. Self-Critique achieves average AUC improvements of up to 30% over baselines, and up to 55% on dual-contamination scenarios, with consistent gains across PPO, GRPO, and DAPO.

Main Findings

• Likelihood-based detectors fail in the RL phase. PPL, Min-K%, and Min-K%++ hover near AUC 0.5 (random guess) because RL optimizes rewards rather than token likelihood, erasing the low-perplexity signals these detectors rely on.
• Entropy is a more informative signal than likelihood for RL. When the same probing mechanisms are used, entropy-based variants (Entropy-Temp, Entropy-Noise) outperform their log-probability counterparts (CDD, Recall), confirming that RL leaves a distinctive entropy fingerprint.
• Active self-critique probing is the strongest strategy. Contaminated samples remain locked into their original reasoning path even when instructed to deviate, producing highly similar entropy curves; clean samples diverge more freely.
• Detection is algorithm-agnostic. Self-Critique attains the best AUC under PPO, GRPO, and DAPO on the K&K task, suggesting the underlying signal (path dependency) is a general property of RL post-training rather than an artifact of one optimizer.
• Top-K entropy approximation is sufficient. Even with K=3, AUC drops only marginally (variance on the order of 10⁻⁵), making the method practical for APIs that expose only top-K token probabilities.
• Dual-stage contamination can be disentangled. When pre-training contamination and RL contamination co-occur (e.g., on GSM8K), filtering out the items with the strongest pre-training signal lets Self-Critique identify RL-phase memorization far more clearly — performance improves as pre-training contamination decreases.
• Consistent results across model families. Gains hold on Qwen2.5 (0.5B/3B/7B/7B-Math), DeepSeek-Math-7B-Instruct, and Llama-3.1-8B-Instruct.

Methodology in Plain English

The approach rests on one observation: RL post-training makes a model converge onto a narrow, high-reward reasoning path — a phenomenon called policy collapse — and this collapse is much stronger for problems the model actually trained on.

1. Get a baseline. Ask the model to solve a problem using greedy decoding, and record the token-level entropy at every position of the generated answer. This entropy sequence acts as a fingerprint of how uncertain the model was along that path.
2. Force a revision. Feed the model its own answer back, along with a meta-prompt asking it to critique and produce a different solution. Record the entropy sequence for this second response.
3. Compare the fingerprints. Compute a length-penalized cosine similarity between the two entropy sequences. Contaminated samples resist deviation — the two sequences are nearly identical. Clean samples explore more, so the sequences diverge.
4. Score and threshold. High similarity means high contamination likelihood. AUC is used to evaluate without committing to a threshold.

For evaluation, the team built RL-MIA by deliberately injecting 50% of each benchmark's items into an RL training corpus (base: OpenR1-Math-46K for AIME) and training models with VeRL on 8×A100 GPUs. This gives ground-truth labels for which samples were seen during RL, enabling controlled measurement.

Why This Matters

Impact on research. Reported benchmark scores for reasoning models are often taken at face value. This work shows that a large class of contamination — the RL post-training phase, which is now central to producing frontier reasoning models — is undetectable by the tools the community currently uses. The paper provides both a corrective method and a benchmark, which enables reproducible research on a problem that previously could not even be measured.

Real-world applications:

• Leaderboard integrity. Benchmark maintainers can screen submissions for RL-phase contamination before publishing rankings.
• Model auditing and compliance. Regulators and enterprise buyers can verify vendor claims that evaluation data was held out from all training stages, including RL.
• Data curation pipelines. Teams training reasoning models can identify which evaluation items leaked into RL data and re-run evaluations on clean subsets.
• Evaluation design. Benchmark creators can release post-cutoff or synthetic datasets (like AIME 2025, K&K, SAT) that resist contamination, informed by what the detectors can and cannot catch.

Industry relevance. Every lab training reasoning models with RLVR faces this problem, and every downstream consumer of published benchmark numbers is affected by it. The method's compatibility with top-K probability APIs makes it deployable even against closed-source models.

Future Directions

• Detection under limited API access. The current method needs token-level probabilities. Extending it to purely text-based observations (e.g., sampling many responses and measuring semantic divergence) would broaden applicability to fully black-box models.
• Standardizing RL contamination benchmarks. RL-MIA is a first step; the community may need larger, multi-domain versions covering code, agents, and multimodal reasoning, with varied injection rates and realistic mixtures.
• Detecting partial or indirect contamination. RL training may leak information through related problems, paraphrases, or reward-model training data rather than direct inclusion. Whether entropy probing catches these softer forms is unknown.
• Combining signals. Fusing entropy similarity with likelihood cues from pre-training detectors could yield a unified detector that handles dual-stage contamination without the manual subsetting the paper currently requires.
• Understanding policy collapse more mechanistically. A deeper account of why contaminated samples resist self-critique — versus clean samples that also show entropy collapse — could lead to more efficient or cheaper detectors.

Target Audience

Researchers and engineers working on LLM evaluation, benchmark design, or reasoning-model training will get the most value. It is also relevant to practitioners building RL post-training pipelines (PPO, GRPO, DAPO), auditors assessing model evaluation claims, and anyone studying membership inference or training-data leakage. Readers without background in RL post-training or entropy analysis will need to consult the cited prerequisites, but the core intuition — models memorize what they were rewarded for — is accessible without deep math.

Authors’ abstract

Data contamination poses a significant threat to the reliable evaluation of Large Language Models (LLMs). This issue arises when benchmark samples may inadvertently appear in training sets, compromising the validity of reported performance. While detection methods have been developed for the pre-training and Supervised Fine-Tuning stages, a critical research gap exists for the increasingly significant phase of Reinforcement Learning (RL) post-training. As RL post-training becomes pivotal for advancing LLM reasoning, the absence of specialized contamination detection methods in this paradigm presents a critical vulnerability. To address this, we conduct the first systematic study of data detection within RL post-training scenario and propose Self-Critique. Our method is motivated by a key observation: after RL phase, the output entropy distribution of LLMs tends to collapse into highly specific and sparse modes. Self-Critique probes for the underlying policy collapse, i.e., the model's convergence to a narrow reasoning path, which causes this entropy reduction. To facilitate this research, we also introduce RL-MIA, a benchmark constructed to simulate this specific contamination scenario. Extensive experiments show that Self-Critique significantly outperforms baseline methods across multiple models and contamination tasks, achieving an AUC improvement of up to 30%. Whereas existing methods are close to a random guess for RL-phase contamination, our method makes detection possible.