Probing RLVR training instability through the lens of objective-level hacking

AI summary

Overview

Research area: Reinforcement learning with verifiable rewards (RLVR) for large language models, with a specific focus on training instability in Mixture-of-Experts (MoE) architectures.

Technical level: Advanced. The paper combines formal derivations of policy-gradient objectives with controlled large-scale training experiments, and assumes familiarity with GRPO, importance sampling, and Mixture-of-Experts models.

Scope: The paper proposes and tests a framework called "objective-level hacking" to explain why the training-inference discrepancy grows abnormally during prolonged RLVR training of MoE models.

Authors and affiliations: Yiming Dong, Kun Fu, Haoyu Li, Xinyuan Zhu, Yurou Liu, Jieping Ye, Zheng Wang (Tongyi Lab, Alibaba Group, and School of Physics, Peking University), Lijing Shao (Kavli Institute for Astronomy and Astrophysics, Peking University; National Astronomical Observatories, Chinese Academy of Sciences). Correspondence: wz388779@alibaba-inc.com. arXiv:2602.01103v2 [cs.AI], 12 May 2026.

What This Paper Is About

Prolonged RLVR training can keep improving the reasoning abilities of LLMs, but it is often unstable, and this instability is especially common in Mixture-of-Experts models. The failure mode typically appears as a continuing decline in model performance accompanied by anomalies in training dynamics such as token-level entropy and gradient norms.

Existing work has empirically introduced stabilization strategies without a mechanistic explanation of why instability happens. This paper argues that instability is driven by objective-level hacking: token-level credit misalignment that injects a system-level spurious signal into the optimization objective, which then drives the abnormal growth of the training-inference discrepancy and, eventually, irreversible model collapse.

Key Contributions

1. A new framing of RLVR instability. The authors introduce objective-level hacking as a counterpart to reward hacking. Reward hacking arises from exploitable verifiers; objective-level hacking arises from token-level credit misalignment and appears as system-level spurious signals in the optimization objective itself.

2. A unified formulation of discrepancy growth. The paper constructs a unified formulation showing that multiple distinct sources of instability—initial training-inference discrepancy, token-level clipping, and explicitly injected token-level weight distortion—all reduce to the same structural form: the intended objective plus an added biased term.

3. Mechanistic explanation of a known pathology. The authors trace the origin and formalize the mechanism behind the abnormal growth of the training-inference discrepancy in MoE models, a phenomenon widely associated with instability but previously lacking a mechanistic account.

4. Empirical verification on a 30B MoE model. Through extensive experiments on a 30B MoE model, the paper shows that different forms of objective-level hacking are statistically correlated with anomalous discrepancy growth, and that this growth in turn undermines training stability.

Main Findings

• The effective objective gains a spurious covariance term. When rollout is generated by an inference policy that differs from the training policy, the effective objective becomes the intended objective plus a correction term approximated as the sum over tokens of the covariance between the token's objective contribution and the inverse importance weight. Training drifts toward unintended correlations between these two quantities.

• Importance-sampling correction slows the discrepancy but does not eliminate it. Applying truncated importance sampling (TIS) markedly reduces the growth rate of the training-inference discrepancy, slows the decline of token entropy, and yields a corresponding improvement in validation scores. However, in MoE training the discrepancy is not completely eliminated even with TIS, indicating additional sources.

• Token-level clipping accelerates discrepancy growth, counterintuitively. Stronger token-level clipping is associated with a faster increase in the training-inference discrepancy in MoE training, despite token-level clipping being widely used to maintain stability. Under similar settings, sequence-level clipping does not exhibit the same anomalous growth. The measured clip ratios (Table 1, from Fig. 4) at gradient steps 4, 200, and 400 are: Token (low) 1.50×10⁻³, 1.15×10⁻³, 1.26×10⁻³; Token (middle) 1.92×10⁻³, 1.81×10⁻³, 2.84×10⁻³; Token (high) 2.37×10⁻³, 2.63×10⁻³, 3.49×10⁻³; Seq. 0.153, 0.120, 0.085.

• Injected weight distortion reproduces the pathology. When different weights are assigned to high- and low-probability tokens (with π_low = 0.1 and δ ∈ {3, 2, 1.2}), the anomalous increase in the training-inference discrepancy reappears, and stronger distortions induce faster discrepancy growth. Model performance degrades irreversibly, accompanied by anomalies in token entropy. Even a modest 20% increase in the weight of low-probability tokens is sufficient to trigger the growth. Reducing that weight also reproduces the growth (Appendix B.2), suggesting the anomaly is driven by weight distortion itself rather than by increasing weight specifically.

• Bias, not variance, is the main driver. Injecting unbiased token-level weight noise sampled independently from a Gaussian distribution N(1, σ²) does not trigger an abnormal increase in the training-inference discrepancy. The authors conclude that biased distortion is the primary driver, because biased objective-level spurious signals can be persistently exploited.

• Objective-level signals can be monitored directly. Using J ≡ Σ Â_{i,t} × (φ_{i,t} − 1) as an approximation to the spurious objective term, the authors show that weight distortion triggers abnormal growth in this signal, providing direct numerical evidence that spurious objective-level signals intervene in optimization.

• Instability forms a positive feedback loop. Low-probability tokens show wider dispersion from an importance weight of 1, and their importance weight consistently decreases as training progresses due to statistical effects induced by survival bias. This divergence exacerbates objective-level hacking, which further grows the discrepancy. The loop explains why degradation is often irreversible: switching to earlier checkpoints or altering training data batches fails to prevent collapse.

• MoE models grow the mismatch far faster than dense models. Extending the analysis to dense models using DeepSeek-R1-Distill-Qwen-7B and the Skywork-OR1-RL-Data training set, the authors find the same qualitative behavior: abnormal mismatch growth, mitigation by TIS and sequence-level clipping, and growth induced by injected token-level distortion. But dense models show only a slight increase, from 1.059×10⁻² to 1.081×10⁻², while MoE models increase from 1.418×10⁻² to 2.664×10⁻² over the same range of gradient steps. The paper attributes the faster MoE growth to expert activation inconsistency, where different experts can be activated during training and inference.

Methodology in Plain English

The research proceeds in three layers.

Theory. The authors start from the GRPO objective, which optimizes a clipped, group-normalized advantage weighted by a per-token importance ratio. They note that in real RL frameworks, rollout generation and model training use different implementations, so responses are actually sampled from an inference policy rather than the training policy. By comparing the intended expectation with the actual expectation, they derive that the effective objective equals the original objective plus a correction term that behaves like a covariance between each token's weighted advantage and its inverse train/inference weight ratio. They then repeat this derivation for token-level clipping, showing that it also adds a biased term because clipped tokens are effectively given zero weight. This yields a unified form: the distributed objective equals the intended objective plus a distortion term arising from token-level credit misalignment.

Measurement. The authors use the standard deviation of the token-level training/inference importance weight ρ_{i,t} over all tokens at a step as a global "mismatch" metric for the training-inference discrepancy.

Experiments. Training is conducted on the Qwen3-30B-A3B model using the DAPO-Math-17k dataset, with AIME24 used as validation to ensure reproducibility. The implementation uses the verl framework with vLLM as the inference backend and Megatron as the training backend, on 4 nodes × 8 A100 GPUs. At each training step, 128 problems are sampled with 16 responses per problem, followed by 4 parameter update steps, an off-policy setting; maximum response length is 8K. For GRPO with token-level clipping, left and right clipping ranges are 0.2; for GSPO with sequence-level clipping, left and right ranges are 3e-4 and 4e-4 respectively.

The authors then run four families of comparisons: vanilla token-level clipping versus the same setting with TIS correction; varying strengths of token-level clipping using the decoupled clipping strategy from DAPO (left range fixed at 0.2, right range over {0.2, 0.24, 0.28}), against GSPO as a sequence-clipping baseline; active injection of token-level weight distortion into a sequence-clipping objective; and an unbiased Gaussian variance injection as a control. Finally, they replicate the design on a dense model with a different training set and compare mismatch dynamics between dense and MoE settings.

Why This Matters

The paper reframes instability as an optimization-objective problem rather than a purely infrastructural artifact. It suggests that mitigating training-inference mismatch at the infrastructure level is not sufficient—the biased objective term must also be addressed. It also explains why sequence-level aggregation algorithms appear more stable in MoE training, and why collapse is often irreversible.

Real-world applications:

• Long-horizon RLVR training pipelines for reasoning models, where knowing which tokens and which clipping choices amplify discrepancy growth can extend the useful training horizon before collapse.
• MoE model deployment and post-training, where the paper's finding that different experts may be activated in training versus inference makes these architectures especially vulnerable to mismatch growth.
• RL framework engineering, since the discrepancy originates from differing implementations of rollout generation and model training, pointing to routing replay, higher numerical precision in selected components, and batch-invariant attention as mitigations.
• Diagnostics and monitoring, because the surrogate objective signal J provides a directly computable quantity that correlates with instability.

Industry relevance: Training runs at the scale studied here are expensive, and collapse wastes compute. A framework that identifies a causal driver and offers a cheap monitoring signal has direct relevance to teams training large reasoning models, particularly those using MoE architectures and token-level clipping objectives.

Future Directions

• Designing algorithms that avoid the dual effects of token-level modulation. The authors suggest that algorithms targeting MoE models should consistently avoid token-level credit misalignment to protect the relatively sensitive inference behavior of these models. What specific objective forms achieve this without sacrificing the stabilization benefits of clipping remains open.

• Reducing expert activation inconsistency. The paper hypothesizes that faster mismatch growth in MoE models stems from different experts being activated during training and inference. Whether rollout routing replay or related techniques can close this gap at acceptable systems cost is not resolved.

• Extending beyond single-cause explanations. The authors explicitly state that RLVR training is a complex multi-system interaction process and that their theoretical explanation does not represent the full picture of model collapse. Model choice, training data, rollout engine, and even hardware configurations may each induce different forms of instability.

• Characterizing the boundaries of the feedback loop. The positive feedback loop between discrepancy and objective-level hacking explains irreversibility, but the conditions under which a run enters the loop versus recovers are not fully specified. The paper notes that the A100 is a non-Hopper architecture GPU with relatively lower numerical precision, leaving open how much numerical precision contributes relative to structural misalignment.

Target Audience

This paper is most valuable to researchers and engineers working on reinforcement learning for large language models—particularly those training or post-training MoE models at scale, and those designing RLVR algorithms and loss aggregation schemes. It is also relevant to ML systems engineers responsible for the rollout and training infrastructure that produces training-inference mismatch, and to researchers studying training stability, interpretability of training dynamics, and model collapse. Readers without a background in policy-gradient methods, importance sampling, and Mixture-of-Experts architectures will find the theoretical sections demanding.

Authors’ abstract

Prolonged reinforcement learning with verifiable rewards (RLVR) has been shown to drive continuous improvements in the reasoning capabilities of large language models, but the training is often prone to instabilities, especially in Mixture-of-Experts (MoE) architectures. Training instability severely undermines model capability improvement, yet its underlying causes and mechanisms remain poorly understood. In this work, we introduce a principled framework for understanding RLVR instability through the lens of objective-level hacking. Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective. Grounded in our framework, together with extensive experiments on a 30B MoE model, we trace the origin and formalize the mechanism behind a key pathological training dynamic in MoE models: the abnormal growth of the training-inference discrepancy, a phenomenon widely associated with instability but previously lacking a mechanistic explanation. These findings provide a concrete and causal account of the training dynamics underlying instabilities in MoE models, offering guidance for the design of stable RLVR algorithms.