ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning

AI summary

Overview

Research area: Reinforcement learning for large language model reasoning — specifically reinforcement learning from verifiable rewards (RLVR), off-policy importance weighting, and the clipped surrogate objectives used by GRPO and GSPO.

Technical level: Advanced. The paper derives its method from Bregman divergences, α-divergence variational objectives, KL projection, and exponential tilting, and applies it to sequence-level importance ratios.

Scope: The paper diagnoses a sign-dependent "gradient starvation" failure in clipped policy optimization when rollouts are reused across updates, proposes a smooth two-branch reshaping kernel called ReSPO, and evaluates it against GRPO, GSPO, and VESPO on math reasoning with Qwen3-1.7B-Base and Qwen3-30B-A3B-Base.

What This Paper Is About

Training reasoning models with RLVR is expensive because generating responses is slow, so practitioners reuse the same rollouts across several policy updates. That makes the training data progressively off-policy, and the importance weights that correct for this mismatch behave badly at their extremes. The authors show that hard clipping — used by both GRPO at the token level and GSPO at the sequence level — misallocates learning signal in a sign-dependent way: it suppresses under-generated positive responses at the low-importance-weight tail while letting severely over-generated negative responses with unbounded weight dominate the high-ratio tail. ReSPO replaces clipping with a smooth, sign-dependent sequence-level kernel designed so that the positive branch keeps a nonzero weight as the ratio goes to zero, while the negative branch vanishes at both tails.

Key Contributions

1. Identification of sign-dependent gradient starvation. The authors formalize two opposite failures of clipped objectives inside the unclipped region: positive responses with importance weight w_t ≪ 1 receive almost no gradient (a weight of w_t = 0.01 receives 100 times less weight than w_t = 1), while negative responses with w_t ≫ 1 remain unclipped and grow without bound, potentially overshadowing other tokens. They argue the same failure carries over to GSPO's length-normalized sequence ratio.

2. A variational derivation of a reshaping kernel family. ReSPO interprets replacing the raw sequence importance weight W with a kernel φ(W) as a change of measure, then solves a dual-proximity objective (1−β)D_f(τ‖μ) + βD_f(τ‖π) under a power-Bregman generator f(x) = 1/(α(α−1)) x^α. This yields the power-mean kernel φ₀(W) = [(1−β) + βW^{α−1}]^{1/(α−1)}, which generalizes VESPO's KL-based kernel φ_KL(W) = W^β (the α → 1 limit). A KL projection under moment constraints then produces a smooth exponential tilt, φ(W) = φ₀(W)·exp(λ(1 − φ₀(W))).

3. A sign-dependent two-branch design with tail guarantees. The paper shows no single value of α satisfies both sign requirements. For α > 1, φ(0) > 0 and φ(∞) = 0 (matching the positive branch); for 0 < α < 1, φ(0) = 0 but φ(∞) > 0; at α = 1, φ(0) = φ(∞) = 0, matching the negative branch. ReSPO therefore uses α⁺ = 2 for positive advantages and the α⁻ = 1 limiting form for negative advantages, with β⁺ = β⁻ = 0.5, λ⁺ = 2, λ⁻ = 2. The resulting kernels are φ⁺(W) = ((1+W)/2)·exp(1−W) and φ⁻(W) = √W·exp(2(1−√W)), which match in value and slope at W = 1.

4. Empirical validation under controlled rollout reuse. ReSPO is compared to GRPO, GSPO, and VESPO on DAPO-MATH-17k under matched data and optimizer settings across N ∈ {8, 16, 32} reuse ratios, with full-split evaluation on six benchmarks. The abstract reports that ReSPO attains a higher late-stage training score than every clipped baseline in all six model–N settings and a higher score than VESPO in five of six.

Main Findings

• Tail behavior separates the methods. The paper's kernel-shape table reports tail limits as W → (0, ∞): clipping gives (0, 0) for the positive branch and (0, ∞) for the negative branch; VESPO gives (0, 0) and (0, 0); ReSPO gives (e/2, 0) and (0, 0). ReSPO's positive kernel peaks at φ⁺(0) = ½e ≈ 1.36 and decreases strictly for W > 0, and its negative kernel peaks at W* = 1/4 with value e/2 and vanishes on both tails.

• Improved early training in five of six settings. ReSPO has the highest observed early peak (within the first 256 policy updates) in five of six model–N settings. Its only shortfall is 30B at N = 8, where its peak is 1.0 pp below GRPO. For N ∈ {16, 32}, its early-peak differences from the highest baseline are 1.0 and 3.0 pp on 1.7B and 4.2 and 5.4 pp on 30B.

• Higher late-stage training score. ReSPO has the highest observed last-128 mean in five of six settings. On 1.7B at N = 16 it is 0.4 pp below VESPO, and each central value lies within the other's temporal range. Its late mean exceeds the highest clipped baseline in all six settings by 4.2–8.4 pp, with non-overlapping temporal ranges; at N = 32 the differences reach 7.9 pp on 1.7B and 8.4 pp on 30B.

• Reported late-mean values. For Qwen3-1.7B, ReSPO's last-128 means are 0.286 ± 0.016 (N = 8), 0.264 ± 0.010 (N = 16), and 0.276 ± 0.004 (N = 32); VESPO's are 0.275 ± 0.016, 0.268 ± 0.009, and 0.230 ± 0.004. For Qwen3-30B-A3B, ReSPO's are 0.595 ± 0.016, 0.563 ± 0.021, and 0.588 ± 0.007; the highest baselines are 0.538 (GRPO and VESPO at N = 8), 0.531 (VESPO at N = 16), and 0.523 (VESPO at N = 32).

• Evaluation advantage grows with rollout reuse. At N = 8 the macro averages are similar relative to their evaluation sampling uncertainty; the bolded best average at N = 8 is GSPO's 0.497 ± 0.018 on 30B and VESPO's 0.353 ± 0.015 on 1.7B. For N ∈ {16, 32}, ReSPO has the highest observed mean in all four model–N settings, exceeding the highest baseline by 0.8 and 1.3 pp on 1.7B and by 3.2 and 1.9 pp on 30B. It has the highest value in 9 of the 12 model–benchmark comparisons at N = 16 and 8 at N = 32, and the highest AIME25 and AMC23 values in every model–N setting. Averaged over AIME25, AIME24, and AMC23 on 30B, its differences from the highest baseline are 6.3, 5.8, and 3.4 pp for N ∈ {8, 16, 32}.

• Advantage persists within response-length bins. Using 10 quantile-based length bins shared across methods and benchmarks, ReSPO has the highest pooled accuracy in 24 of the 30 bins. At N = 32 it exceeds the highest baseline from Q1 through Q7 by 8.4, 16.0, 24.4, 21.6, 20.8, 15.1, and 8.5 pp respectively. Bin Q10 contains responses at the 16,384-token cap.

• On-policy equivalence. For an on-policy update, W = 1 and both branches reduce to unit weight, and the two branches match in value and slope at W = 1.

• Difference from VESPO. VESPO's exponential factor acts on the raw weight W, giving W^β exp(λ(1−W)), whereas ReSPO's tilt acts on the reshaped weight, giving W^β exp(λ(1−W^β)) in the α → 1⁺ limit.

Methodology in Plain English

The authors start from a conceptual argument rather than a benchmark. First, they ask what four sign-and-tail combinations require: a positive response whose probability is shrinking should be reinforced, a positive response already reinforced should be damped, a negative response already suppressed should stay suppressed, and a negative response being amplified should be suppressed hard. That gives two boundary conditions — the positive kernel must be positive at zero, and all other tails must go to zero.

Second, they obtain a family of kernels that could satisfy these conditions. They treat the reshaped weight as an unnormalized measure τ = μ·φ(W) sitting between the behavior policy μ and the target policy π, and solve a trade-off objective controlled by β ∈ (0,1) under a power-Bregman generator. The solution is a power-mean interpolation whose tails depend on α. Because an unconstrained kernel is not enough for variance control, they project it onto moment constraints using a KL divergence, which turns the constraint into a smooth multiplicative exponential tilt with multiplier λ instead of a hard cutoff.

Third, they pick α by the sign of the advantage because no single α meets both sets of requirements, then choose β = 0.5 for a symmetric interpolation and choose λ by matching the two branches' values and slopes at the on-policy point W = 1. This yields the two closed-form kernels used in training. The kernel is applied to the unnormalized sequence ratio W (not GSPO's length-normalized form, which the authors argue distorts the geometry) and is detached with a stop-gradient so it acts as a pure per-sequence gradient scaling coefficient in a REINFORCE-style estimator. A numerically stable log-space form and the full update algorithm are given in the appendices.

Empirically, all experiments use verl with asynchronous vLLM rollouts, mini-batch size M = 32, G = 8 rollouts per prompt, and the GRPO advantage estimator. The dense model uses FSDP with AdamW; the MoE model uses Megatron with Adam; both use a learning rate of 10⁻⁶ and no KL penalty. Training is on DAPO-MATH-17k with a strict-box verifier. The global batch size is N·M, split into N consecutive mini-batches, so larger N means more updates per π_old and more drift. Every run is fixed at

Authors’ abstract

Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy. We identify a sign-dependent gradient starvation problem in clipped policy optimization: clipping suppresses under-generated positive responses at the low-importance-weight tail while permitting severely over-generated negative responses to dominate the high-weight tail. To address this, we propose ReSPO (Reshaped Sequence Policy Optimization), which replaces clipping with a smooth, two-branch sequence-level kernel derived from an $α$-divergence variational objective and an exponential variance-control tilt. The positive branch preserves a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses. We demonstrate that ReSPO effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail. On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under a rollout reuse, validating our approach on importance-weight tail control in off-policy learning.