Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

AI summary

Overview

Research area: Mechanistic interpretability of reinforcement learning with verifiable rewards (RLVR) in large language models, specifically the interaction between memorization and reasoning in Qwen2.5-Math models.

Technical level: Advanced. The paper assumes familiarity with Transformer internals (residual streams, MLP key-value memories, path patching, Logit Lens, Jensen-Shannon divergence) and uses Neural Differential Equations as an analytical framework.

Scope: The paper traces how spurious RLVR rewards mechanically reconfigure the internal layers of Qwen2.5-Math models to retrieve memorized answers rather than reason, and shows that this circuit can be causally steered.

What This Paper Is About

Prior work showed that Qwen2.5-Math models gain benchmark accuracy even when trained with random, format-only, or intentionally incorrect rewards, and that these gains likely come from data contamination rather than improved reasoning. This paper asks the mechanistic question left open by that work: if the contaminated answers were already stored in the base model during pretraining, why was initial accuracy low, and how does training on incorrect rewards unlock them? The authors localize the answer to a specific circuit inside the network and demonstrate they can turn the resulting performance up or down by hand.

Key Contributions

1. Discovery of the Anchor–Adapter circuit. The authors causally localize the mechanism behind spurious gains into a Functional Anchor in middle layers (L18–L20) that triggers retrieval of memorized answers, and Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal rather than store new knowledge.

2. Empirical validation of contamination. Through reset and keep-only ablations, the authors show these circuits are dataset-dependent: manipulating them eliminates gains on contaminated benchmarks while leaving general reasoning and a leakage-free benchmark intact.

3. Causal steering via MLP intervention. By scaling the keys of specific MLP neurons identified within these layers, the authors bidirectionally steer performance, artificially inflating or suppressing contamination-driven accuracy, which confirms the causal role of those neurons in shortcut activation.

4. A probe-gated shortcut audit. A selective variant of the intervention scores each prompt with a Layer-19 leakage probe and intervenes only above a conservative threshold, converting the method into a counterfactual test of whether an RLVR success is shortcut-dependent.

Main Findings

• Perplexity Paradox. On contaminated data (MATH-500), Qwen2.5-Math's answer-only perplexity decreases progressively across incorrect-reward checkpoints at step 0, 50, 100, and 150, while full-text perplexity increases. LLaMA-3.1-8B and OLMo-2-1124-7B show rising perplexity in both metrics, and on leakage-free LiveMathBench Qwen's perplexity remains at the highest level. The authors read this divergence as the model sacrificing prompt-side coherence to form a prompt-to-memorized-answer mapping.

• Path patching shows MLPs, not attention, carry the shortcut. Replacing base-model activations with spurious-RLVR activations recovers significantly more accuracy when patching MLPs than attention heads. In Qwen, recovery stays consistently high across early layers, holds at Layers 18–20, then drops precipitously at Layer 21. LLaMA shows no comparable recovery pattern.

• Counterfactual JSD separates structure from function. For the gated MLP, JSD scores for W_up and W_gate rise to a peak at Layers 21–22 and then decline, while W_down maintains a sustained upward trend converging to a high plateau. The authors interpret this as parametric updates encoding memorized features at L21–22 and then permanently rotating the feature space, making later MLP outputs structurally incompatible with the base model's remaining layers. LLaMA instead shows monotonically increasing JSD across all components with no peak-reversal.

• Logit Lens case study. On a leakage sample from MATH-500 generated at temperature 0.7, the successful trajectory shows the MLP at Layer 19 introducing a precursor signal, a temporary probability drop at Layers 21–22, and the MLP at Layer 23 aggressively injecting the correct answer token "4". In the failure trajectory (output "3"), the MLPs at Layers 23–25 still attempt to inject "4" but the Layer 19 priming signal is weaker, so the residual stream drifts to the incorrect token.

• Neural Differential Equation analysis locates the bifurcation. PCA-projected latent trajectories of leakage and stable samples overlap in early layers and bifurcate in the middle layers. Separation force peaks precisely at Layers 18, 19, and 20, while velocity difference is minimal at L19 and increases in later layers, consistent with the anchor setting direction and adapters amplifying magnitude. Linear probe AUC-ROC peaks at Layer 20.

• Anchor layers are more critical than adapter layers. Resetting the Anchor drops leakage accuracy by more than 10% on contaminated datasets (MATH-500: 98% to 86%; MinervaMath: 88% to 72%), while Adapter Reset produces smaller drops (Δ = −6% and −10%). The remaining 86% is explained by pre-existing contamination: the base model already reaches 78% on the MATH-500 leakage subset.

• Neither layer group alone is sufficient. On MinervaMath, Keep Only Adapter yields 32% accuracy versus 50% for the base model, Keep Only Anchor reaches 42%, and Keep Both improves to 46%, indicating a collaborative circuit. Randomizing the Anchor collapses MATH-500 accuracy to 34%, and Keep Only Adapter reaches 80%.

• Clean data and stable samples are insensitive. LiveMathBench accuracy remains stable at 70% across all ablation conditions, and stable sample accuracy stays nearly identical to the RLVR baseline across all reset experiments.

• Within-family evidence at smaller scale. On a 38-case MATH-500 leakage subset from Qwen2.5-Math-1.5B using the same Anchor (L18–20) and Adapter (L21–22) partition, Anchor Reset drops accuracy from 39.5% to 2.6%, Keep Only Adapter falls to 0%, and the largest mean FFN drop occurs at L18. Path patching is again MLP-dominant, with FFN AUC 0.0789 versus attention AUC 0.0498 (paired sign-flip p = 6.99965 × 10⁻⁴).

• Neuron-level steering is bidirectional. Task-relevant neurons are identified by a relevance score combining key activation magnitude with semantic overlap with answer tokens, selecting the top k = 10 neurons per layer. These neurons peak in activation at L19, L21, and L24. Layer 18 shows maximal sensitivity (−3.8% under suppression, +4.4% under amplification), while Layer 25 shows uniform degradation across all factors. On the leakage-free dataset, steering yields no systematic pattern. Amplification (α = 3.0) causes a special formatting token to reach top-1 at Layer 22 one layer earlier, and suppression (α = 0.2) delays trigger activation. For one case (Question 213, ground truth "2"), amplification produces a qualitatively different trajectory in which a dominant token holds high probability from the middle layers and the correct answer "2" emerges abruptly only in the final two layers.

• Probe-gated audit is selective. Using Layer 19, the top-50 probe-ranked neurons, and a suppression factor α = 0.5, the gate activates on 61.7% of the MATH-500 wrong→right subset (79/128) and removes 19 RLVR-only successes, reducing accuracy from 100.0% to 85.16% (exact McNemar p = 3.81 × 10⁻⁶). It fires on 0/30 AIME-2025 and 0/100 LiveMathBench examples, leaving their accuracies identical at 30.0% and 21.0% respectively. OlymMATH-en-easy is unchanged at 7.0%, and OmniMATH100-rule changes by only one point, from 36.0% to 35.0% (paired test p = 1.0).

Methodology in Plain English

The authors start from a puzzle: a base model already contains contaminated test answers, yet scores poorly until trained with meaningless rewards. They first measure how uncertainty evolves during training by separating perplexity on the answer text from perplexity on the full prompt-answer text, tracking four checkpoints (step 0, 50, 100, 150) of incorrect-reward training.

To find where the shortcut lives, they compare the base model with the spurious-RLVR-tuned checkpoint on matched questions, splitting examples into "leakage" samples (wrong before RLVR, correct after) and "stable" samples. They then run three complementary probes. Path patching swaps activations between the two model versions at specific layers and measures how much accuracy is recovered, which tells them which components causally matter. Logit Lens reads out the model's intermediate token predictions at every layer, which shows when a specific answer token appears in the residual stream. Counterfactual JSD replaces each MLP sub-matrix (W_up, W_gate, or W_down) with a neutral or random substitute and measures how much the output distribution shifts, which reveals where the largest structural changes occurred.

They then reframe the layer-by-layer updates as a continuous trajectory using Neural Differential Equations, fitting a two-layer MLP to the discrete residual updates and computing "separation force" and "velocity difference" between leakage and stable samples, plus linear probes at each layer to test how separable the two groups are.

Finally, they move from observation to intervention. They score individual MLP neurons by how strongly their key activations fire and how much their value vectors overlap with answer tokens, take the top 10 per layer, and multiply those key activations by a scaling factor α during inference. A gated version of this intervention uses a Layer-19 probe and a threshold set at the 99th percentile of negative-control prompts to intervene only on suspicious prompts.

Why This Matters

For research, the paper supplies a mechanistic account of a widely observed but poorly explained phenomenon: RLVR benchmark gains that do not come from improved reasoning. It reframes data contamination as an activatable circuit with identifiable components rather than a diffuse property of the training set, and it offers a causal, auditable test for shortcut dependence. The work also connects behavioral "over-memorization" signatures (high task accuracy with degraded language modeling) to a concrete layer-level mechanism.

Real-world applications:

• Benchmark auditing and evaluation. The probe-gated intervention can be run as an offline audit to determine whether a reported accuracy gain survives when contamination-sensitive circuits are suppressed.
• Model release and due diligence. Organizations comparing checkpoints could use the anchor/adapter partition to flag whether improvements are localized to contaminated retrieval rather than general capability.
• Safety-critical deployment. In domains where a model must reason rather than recall, the audit identifies prompts that trigger memorized shortcuts before deployment.
• Post-training diagnostics. The perplexity divergence offers a cheap macroscopic signal that something is being memorized during a training run, detectable without layer-level access.

Industry relevance: The methods operate at inference time on existing checkpoints and require no retraining, which makes them practical for teams shipping RLVR-tuned models. The demonstrated ability to amplify contaminated performance by roughly 4% at a single layer also matters for anyone validating that a model's published benchmark numbers reflect genuine capability.

Future Directions

• Whether the Anchor–Adapter circuit generalizes beyond the Qwen2.5 family. The authors frame their 1.5B results as within-family evidence rather than universality, and LLaMA-3.1-8B and OLMo-2-1124-7B show no equivalent mechanism, leaving open what architectural or data conditions produce the circuit.
• Whether the same localization applies to non-mathematical domains. All experiments use mathematical reasoning benchmarks, and the authors do not report whether coding or other deterministic domains show the same layer structure.
• How the anchor mechanism relates to the single steerable reasoning/memory direction reported by prior work, since the paper demonstrates layer-level and neuron-level control rather than identifying one global direction.
• Whether the gated audit can be made fully reliable as a mitigation rather than a diagnostic. The gate fires on 0/30 AIME-2025 and 0/100 LiveMathBench cases, so the authors explicitly characterize it as an offline shortcut-dependence audit rather than a blanket reasoning improver.

Target Audience

This paper is most useful to mechanistic interpretability researchers, RLVR and post-training practitioners who need to distinguish genuine capability gains from contamination artifacts, and evaluation engineers responsible for benchmark integrity. Readers need working knowledge of Transformer internals, but the high-level framing around the Perplexity Paradox and the ablation results is accessible to anyone who tracks RLVR performance claims.

Authors’ abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18-20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering-artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models. Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts.