DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning

AI summary

Overview

• Research area: Reinforcement learning for large language models, specifically reinforcement learning with verifiable rewards (RLVR) applied to mathematical reasoning.
• Technical level: Advanced. The paper assumes familiarity with policy-gradient methods (REINFORCE, PPO), importance-sampling ratios, trust regions, and LLM post-training pipelines.
• Scope (one sentence): The paper introduces DISPO, a REINFORCE-style RLVR algorithm that decouples the clipping of importance-sampling weights for correct versus incorrect responses into four tunable regimes, and shows it improves both training stability and final accuracy over DAPO and CISPO on mathematical reasoning benchmarks.

What This Paper Is About

Reinforcement learning with verifiable rewards has become a standard way to improve LLM reasoning, but existing algorithms force a trade-off: PPO-style methods such as GRPO and DAPO are stable yet learn slowly because of their trust-region constraints, while REINFORCE-style methods such as CISPO learn faster but can suddenly collapse mid-training. DISPO's goal is to break this trade-off by giving the practitioner separate control knobs over how strongly the model is pushed on correct responses versus incorrect responses, and on tokens whose importance-sampling weight has drifted above 1 versus below 1.

Key Contributions

1. A decoupled clipping objective. DISPO modifies off-policy REINFORCE by assigning four separate clipping bounds to the importance-sampling weight, conditioned on the sign of the advantage (correct vs. incorrect response) and on whether the weight is above or below 1, yielding four controllable policy-update regimes.
2. A systematic characterization of those four regimes through ablations. The authors isolate each regime on Qwen3-14B-Base and document its distinct effect on token entropy, response length, and accuracy, including the two distinct catastrophic failure modes that appear when incorrect-response regimes are over-constrained.
3. A diagnosis of previously reported CISPO collapses. The paper attributes sudden CISPO failures to insufficient relaxation of the incorrect-response clipping regimes, linking them to repetition collapse and vanishing response lengths, and reports reproducing both failure patterns in their own CISPO setup.
4. State-of-the-art empirical results across models and benchmarks. DISPO reaches 61.04% on AIME'24 with Qwen3-14B-Base, versus 55.42% for CISPO and 50.21% for DAPO — a 10.83 percentage-point gain over DAPO — with consistent gains on Qwen3-8B-Base and Qwen3-30B-A3B-Base (MoE, 3.3B activated parameters) and on AIME'25, AMC'23, MATH-500, and Minerva.

Main Findings

• DISPO sets a new high score on the reported benchmarks with Qwen3-14B-Base. AIME'24: 61.04 (DISPO) vs. 55.42 (CISPO) vs. 50.21 (DAPO) vs. 3.96 (starting checkpoint); AIME'25: 45.83 vs. 40.83 vs. 38.96 vs. 1.67; AMC'23: 92.03 vs. 89.84 vs. 87.66 vs. 22.19; MATH-500: 94.61 vs. 93.15 vs. 91.89 vs. 73.91; Minerva: 46.78 vs. 45.66 vs. 45.22 vs. 36.10.
• Gains hold across model sizes and architectures. With Qwen3-30B-A3B-Base (MoE), DISPO scores 53.75 on AIME'24 (DAPO 45.83, CISPO 48.54) and 89.38 on AMC'23 (DAPO 83.75, CISPO 87.03). With Qwen3-8B-Base, DISPO scores 45.00 on AIME'24 (DAPO 38.33, CISPO 43.54) and 87.66 on AMC'23.
• Regime 1 (correct response, importance weight > 1) drives exploration. Amplification of already-favored tokens in correct responses raises average token-level entropy, and runs with the corresponding clipping parameter above zero improve faster than an online SFT baseline whose entropy stays constant. The paper notes that amplifying gradients for low-reference-likelihood tokens improves token utilization.
• Excessive exploration in Regime 1 causes gradual, not sudden, degradation. Both Regime 1 configurations peak and then decline as the model samples increasingly unlikely tokens that harm reasoning coherence. The more conservative setting (0.28) explores more gradually than the aggressive setting (10), enabling longer training and higher peak accuracy.
• Regime 2 (correct response, importance weight < 1) acts as distillation. Allowing the weight to fall below 1 reduces token entropy — the opposite of Regime 1 — by pruning unreliable tokens even within correct responses, which yields efficiency gains over the SFT baseline.
• Distillation alone is weaker than exploration and best used as a counterweight. Regime 2's standalone gains are modest, and its falling entropy signals premature convergence. When Regimes 1 and 2 operate together, their opposing entropy effects produce a balanced dynamic that outperforms either regime alone.
• Disabling Regime 3 (incorrect response, weight > 1) causes sudden repetition collapse. Setting the corresponding upper clipping bound to 0 caps the weight at 1, weakens gradients for high-probability incorrect tokens, and leads to an early accuracy collapse and response-length spike as the model repeatedly generates the same incorrect tokens. The paper connects this to reported CISPO collapses.
• Disabling Regime 4 (incorrect response, weight < 1) causes length collapse. Setting the corresponding lower clipping bound to 0 removes suppression of already-strong gradients on low-probability incorrect tokens, causing excessive penalization that drives response length toward zero and rapidly deteriorates accuracy.
• Entropy distinguishes DISPO from CISPO. In the AIME'24 learning curves, CISPO loses entropy throughout training while DISPO's entropy increases. DAPO also shows rising entropy but lower accuracy because its token-level PPO clipping discards low-reference-likelihood tokens such as "but", "aha", and "since" that the paper identifies as key entropy drivers.
• Clipping configuration matters enormously for the baselines. DAPO is run with the standard 0.2 / 0.28 bounds. Because CISPO's original bounds were never released, the authors use 1 and 100; they report that either a lower bound below 1 or an upper bound below 100 caused sudden performance collapse in their CISPO runs.

Methodology in Plain English

The setup follows the standard RLVR recipe. For each question drawn from a math dataset, the model samples a group of responses, and a verifier marks each response correct or incorrect with a reward of either 1 or −1. The group's mean and standard deviation turn these rewards into an advantage: positive for correct responses, negative for incorrect responses.

The key quantity in these algorithms is the importance-sampling weight — the ratio between the probability the current model assigns to a token and the probability the frozen reference model assigned to it. This ratio says whether the model has, since the last update, become more or less likely to produce that token. REINFORCE multiplies the loss by this ratio but does not differentiate through it; PPO-style methods clip it and zero out the update once the ratio leaves a trust region; CISPO clips it but still allows gradients outside the trust region.

DISPO changes one thing: instead of one clipping window applied uniformly to all tokens, it uses four windows. Correct responses get one pair of lower/upper bounds; incorrect responses get a different pair. Within each of those, the upper bound controls how much amplification is allowed when the ratio exceeds 1, and the lower bound controls how much suppression is allowed when the ratio falls below 1. All four bounds are treated as tunable hyperparameters.

To understand what each knob does, the authors run controlled ablations on Qwen3-14B-Base. For the two correct-response regimes, they start from an online supervised fine-tuning baseline and switch on one regime at a time (using upper-bound values of 0.28 and 10 for Regime 1, and lower-bound values of 0.2 and 1 for Regime 2). For the two incorrect-response regimes, they start from the full DISPO configuration and disable one regime at a time by zeroing its bound. In every run they track accuracy, token entropy, and response length. All reported scores are averaged over 16 samples per question (Avg@16). Training uses a group size of 16, a mini-batch size of 512, a micro-batch size of 32 (16 gradient updates per mini-batch), dynamic sampling, and an overlong penalty, and the authors state they do not include a KL divergence regularization term. The final DISPO configuration is the lower/upper pair 0.2 and 10 for correct responses, and 1 and 100 for incorrect responses.

Why This Matters

For the research community, the paper reframes the PPO-versus-REINFORCE debate as a problem of clipping design rather than an inherent trade-off. Its regime-by-regime ablation gives a mechanistic account — via entropy and response-length trajectories — of why REINFORCE-style RL collapses, which is a more actionable diagnosis than simply reporting that it does.

The paper's authors list code generation and open-ended dialogue as domains with more nuanced reward structures where DISPO has not been tested, and they note that their study is limited to mathematical reasoning with binary rewards. The real-world applications below are plausible implications of stable, efficient RL post-training, not results the paper demonstrates.

• Mathematical and quantitative tutoring or solving assistants, where the paper's own benchmarks (competition problems such as AIME and AMC) are the target capability.
• Code generation and software agents, where a verifiable pass/fail signal is available from running tests, though the authors explicitly say this is unexplored for DISPO.
• Scientific and engineering reasoning tools, where the paper lists logical deduction and scientific analysis among the reasoning tasks LLMs address.
• Any production RL post-training pipeline that currently stops training early to avoid instability, since DISPO's stated benefit is enabling longer training without collapse.

Industry relevance is direct: the work was performed at Amazon AGI (with the first author's contribution done during an internship there), and the reported efficiency argument — reaching higher accuracy with fewer updates than trust-region methods — speaks to the cost of large-scale RLVR runs. The authors also note that computational constraints capped their experiments at models up to 30B parameters.

Future Directions

• Extending DISPO beyond binary rewards. The authors name code generation and open-ended dialogue as domains with more nuanced reward structures that remain unexplored.
• Adaptive or learned clipping schedules. Determining the four clipping parameters still requires trial and error; the authors propose automatically adjusting them based on training dynamics.
• Detecting and preventing gradual degradation. The paper shows that excessive exploration in Regimes 1 and 2 degrades performance late in training, and suggests entropy regularization or dynamic exploration schedules as possible principled fixes.
• Scaling beyond 30B parameters. The experiments stop at models up to 30B parameters because of computational constraints, leaving scalability to larger reasoning systems as an open question.

Target Audience

This paper is most useful to RL and LLM post-training researchers and engineers who work on RLVR pipelines — particularly those who have hit REINFORCE-style instability or PPO-style slow convergence and are looking for a low-complexity modification they can tune. It is also relevant to practitioners who need to reproduce or debug CISPO-style training collapses, since the paper's ablations offer concrete failure signatures (repetition spikes and shrinking response lengths) tied to specific clipping choices. Readers without a background in policy-gradient methods and importance sampling will find the regime analysis difficult to apply without additional reading.

Authors’ abstract

Reinforcement learning with verifiable rewards has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models particularly in mathematics. Current approaches in this domain present a clear trade-off: PPO-style methods (e.g., GRPO/DAPO) offer training stability but exhibit slow learning trajectories due to their trust-region constraints on policy updates, while REINFORCE-style approaches (e.g., CISPO) demonstrate improved learning efficiency but suffer from performance instability as they clip importance sampling weights while still permitting non-zero gradients outside the trust-region. To address these limitations, we introduce DISPO, a simple yet effective REINFORCE-style algorithm that decouples the up-clipping and down-clipping of importance sampling weights for correct and incorrect responses, yielding four controllable policy update regimes. Through targeted ablations, we uncover how each regime impacts training: for correct responses, weights &gt;1 increase the average token entropy (i.e., exploration) while weights &lt;1 decrease it (i.e., distillation) -- both beneficial but causing gradual performance degradation when excessive. For incorrect responses, overly restrictive clipping triggers sudden performance collapse through repetitive outputs (when weights &gt;1) or vanishing response lengths (when weights &lt;1). By separately tuning these four clipping parameters, DISPO maintains the exploration-distillation balance while preventing catastrophic failures, achieving 61.04% on AIME'24 (vs. 55.42% CISPO and 50.21% DAPO) with similar gains across various benchmarks and models.