Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward

AI summary

Overview

• Research area: Reinforcement learning with verifiable rewards (RLVR) for improving LLM reasoning, with a focus on the exploration–exploitation trade-off in Group Relative Policy Optimization (GRPO).
• Technical level: Advanced. The paper derives explicit bounds on clipping bias and one-step policy entropy change, though the argument and conclusions are readable at a conceptual level.
• Scope: A theoretical and empirical study of how ratio clipping, policy entropy, and spurious (random) rewards interact to produce — or fail to produce — reasoning gains in RLVR, evaluated on Qwen-Math, Llama, and QwQ model families.

What This Paper Is About

Recent work reports a paradox in RLVR: both spurious rewards (random feedback unrelated to ground truth, which discourages exploitation) and entropy minimization (which discourages exploration) have been claimed to improve LLM math reasoning. The paper asks how policy entropy actually relates to performance, and whether spurious rewards genuinely help — or whether reported gains come from clipping bias and benchmark contamination in Qwen-Math models. The authors answer with new theory and experiments spanning multiple model families and sizes.

Key Contributions

1. New theory of clipping bias under spurious rewards. The authors derive explicit bounds on the total upper-clipping correction term and show that, under random rewards, this bias does not constitute a meaningful learning signal. In their numerical evaluation with practical hyperparameters, the raw surrogate term exceeds the clipping correction by a factor of at least 17.15.
2. A one-step policy-entropy-shift formulation. They establish a deterministic link between clipping and policy entropy: clipping systematically reduces entropy and drives the policy toward more deterministic, higher-confidence rollouts. They also derive the unclipped case, where entropy change depends on the skewness of the initial policy (measured by Φ(π_old)).
3. Experiments across multiple model families and sizes. They test Qwen-Math, Llama, and QwQ at 7B, 8B, and 32B scale, including base and distilled variants, reconciling conflicting reports and showing that performance improvements under spurious rewards are robust rather than tied to a single model or dataset.
4. A reward-misalignment model. To explain why spurious rewards can help beyond contaminated settings, they propose a reward-misalignment model, arguing that these gains cannot be attributed to clipping bias or to a causal effect of policy entropy — overturning the view that the effect is confined to potentially contaminated Qwen-Math models.

Main Findings

• Policy entropy decreases under clipped random-reward training. Experiments on Qwen2.5-Math-7B show that with clipping enabled, policy entropy decreases monotonically under random rewards, matching Theorem 4.3.
• Disabling clipping can increase entropy. Without clipping, entropy increased over training in the same setup, and in one case the training trajectory showed an increase in exploration. This shows the direction of entropy change is governed by clipping, not by the random reward alone.
• Clipping activation is rare. The authors report that the clipping activation ratio is usually below 1% in general GRPO training, and never exceeds 0.2% for Qwen2.5-Math-7B, with expected token-level activation 𝔼[I_t] ≈ 0.001.
• Clipping bias is small relative to the raw signal. Using their actual training setup (η = 5×10⁻⁷, ε = 0.2, p₊ = 0.001, G = 16, L = 4096, π_min = 10⁻⁶), they compute M = 3.75, 𝔼[|A|] ≈ 0.967, R_η^max ≈ 1.649, φ(R_η^max) ≈ 0.176, Δ_η⁺ ≈ 0.449, and a ratio of raw surrogate to clipping correction of at least 17.15.
• Enabling clipping did not help performance. In independent trials on Qwen2.5-Math-7B with the MATH500 validation set, enabling clipping led to a decline in validation performance, while disabling clipping often resulted in improvement.
• Entropy change without clipping depends on policy skewness. For a two-armed policy (β, 1−β), Φ(π_old) = 1 + (1−2β)·log(β/(1−β)), which is non-negative exactly when β ∈ [0.176, 0.824]. Less-skewed initializations lose entropy; more-skewed ones gain it.
• Unclipped training can destabilize models with large gradients. For R1-Distill-Llama-8B, unclipped training raised MATH500 validation accuracy from 65.6% to 76.6% within 100 steps, but around step 150 the gradients exploded and performance dropped sharply.
• Clipping acts as regularization, not as a learning signal. Capping per-token likelihood ratios reduces the effective update step size, keeps the policy near its previous distribution, and prevents gradient explosion — but it does not add useful gradient information.
• Gains under spurious rewards extend beyond Qwen-Math. Similar improvements also arise in the Llama and QwQ families, which the authors say cannot be explained by contamination alone.

Methodology in Plain English

The authors combine theory with controlled training runs. On the theoretical side, they start from the standard GRPO objective, write down the clipped per-token surrogate, and split it into a raw term and a clipping-correction term. They then bound the expected magnitude of the clipping correction and compare it to the raw term, showing the correction is negligible under the hyperparameters practitioners actually use. Separately, they model the softmax policy update and expand the change in policy entropy over one step, once without clipping and once with clipping, to isolate what clipping does to entropy. They also prove that under random rewards the advantage is symmetrically distributed around zero, which makes an existing entropy-change approximation fail and motivates their more careful analysis.

On the empirical side, they fine-tune models with random rewards drawn from Bernoulli(1/2), following the hyperparameter configuration of prior work: batch size 128, group size 16, decoding temperature 1.0, clipping ratio 0.2, learning rate 5×10⁻⁷, and KL coefficient 0, using the verl framework and the DeepScaleR dataset. They run multiple consecutive trials with and without clipping, tracking MATH500 validation accuracy, clipping activation fraction, and policy entropy. They also report a reward-misalignment model to explain when spurious rewards can be beneficial.

Why This Matters

Research impact. The paper challenges a widely reported explanation for why random rewards appear to help RLVR — namely upper-clipping bias amplifying high-prior responses in contaminated Qwen-Math models. It replaces that story with a sharper one: clipping controls entropy, and entropy reduction alone is not what produces gains. This reframes several conflicting results in the literature (including work reporting gains from entropy minimization, and work reporting gains only on contaminated benchmarks) as different observations of the same underlying dynamic.

Potential application areas (note: the paper does not report deployed systems or real-world trials):

• Designing RLVR training pipelines for mathematical and STEM reasoning models, where the choice of clipping threshold and group size affects stability and entropy.
• Diagnosing training instability in LLM post-training, since the paper shows a concrete gradient-explosion failure for R1-Distill-Llama-8B without clipping.
• Developing entropy-control strategies that preserve exploration, by combining true and spurious rewards rather than relying on regularization that only slows entropy decay.
• Benchmark-quality auditing, given the paper's discussion of contamination on MATH500.

Industry relevance. Practitioners training reasoning models with GRPO-style algorithms can use the reported clipping activation rates and the bound comparison to check whether clipping is contributing any signal in their own setup — the authors note that the activation rate can be monitored directly during training, making the analysis model-agnostic rather than architecture-specific.

Future Directions

• Combining true and spurious rewards. The authors suggest that spurious-reward setups could be used to actively preserve or increase entropy, complementing existing regularization techniques that only slow entropy decay.
• When do spurious rewards actually help? The proposed reward-misalignment model is offered as an explanation beyond contamination, but the exact conditions under which misaligned rewards improve performance remain an open question the paper raises.
• Entropy as a control knob rather than an objective. Since entropy minimization alone is reported as insufficient for improvement, a natural next step is characterizing what additional condition makes lower entropy beneficial.
• Extending beyond the tested setting. The empirical analysis focuses primarily on MATH500 and on Qwen-Math, Llama, and QwQ families at 7B, 8B, and 32B scale; generalization to other benchmarks, model families, and reward designs is left open.

Target Audience

Researchers and engineers working on LLM post-training and reinforcement learning — particularly those implementing GRPO or similar clipped policy-gradient methods — will benefit most. It is also relevant to readers interested in the exploration–exploitation trade-off and entropy dynamics in sequence-level RL, and to those evaluating claims about reasoning gains from RLVR. Readers wanting the full theoretical detail will need comfort with policy-gradient derivations, importance sampling, and entropy analysis; the conclusions and experimental patterns are accessible without that background.

Authors’ abstract

This paper examines the exploration-exploitation trade-off in reinforcement learning with verifiable rewards (RLVR), a framework for improving the reasoning of Large Language Models (LLMs). Recent studies suggest that RLVR can elicit strong mathematical reasoning in LLMs through two seemingly paradoxical mechanisms: spurious rewards, which suppress exploitation by rewarding outcomes unrelated to the ground truth, and entropy minimization, which suppresses exploration by pushing the model toward more confident and deterministic outputs, highlighting a puzzling dynamic: both discouraging exploitation and discouraging exploration improve reasoning performance, yet the underlying principles that reconcile these effects remain poorly understood. We focus on two fundamental questions: (i) how policy entropy relates to performance, and (ii) whether spurious rewards yield gains, potentially through the interplay of clipping bias and model contamination. Our results show that clipping bias under spurious rewards reduces policy entropy, leading to more confident and deterministic outputs, while entropy minimization alone is insufficient for improvement. We further propose a reward-misalignment model explaining why spurious rewards can enhance performance beyond contaminated settings. Our findings clarify the mechanisms behind spurious-reward benefits and provide principles for more effective RLVR training.