RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

AI summary

Overview

Research area: Post-training of large language models — specifically on-policy distillation (OPD), reinforcement learning with verifiable rewards (RLVR), and self-improvement from a model's own training trajectory.

Technical level: Advanced. The paper assumes familiarity with policy gradients (GRPO/DAPO), KL and Jensen–Shannon divergences, top-K distribution approximation, task arithmetic, and low-rank/linear training-trajectory analyses.

Scope: RISE is a training loop that builds its own distillation teacher by linearly extrapolating the displacement between the current checkpoint and a trailing anchor, then distills that teacher back into the student with per-token supervision, recursively across iterations.

What This Paper Is About

On-policy distillation gives a language model dense, per-token supervision, but it only works as well as its teacher. External teachers suffer from prefix distribution mismatch as the student explores new reasoning paths, and self-distillation teachers conditioned on privileged information (such as a correct solution) are limited by the model's in-context learning ability. RISE sidesteps both by asking what the teacher should be: rather than importing one, it extrapolates the model's own RLVR parameter update to synthesize a stand-in for the model's more capable future self, then distills that back into the student.

Key Contributions

1. Diagnosis and reframing of the OPD bottleneck. The authors identify teacher quality as the central limitation of OPD and propose RISE, which constructs the teacher by extrapolating the model's own RLVR trajectory — requiring no external model and no privileged context. Because the teacher is refreshed every iteration from the student's latest update, distillation becomes a recursive improvement loop rather than a one-shot compression step.

2. A unified construction instantiated in two spaces. The extrapolation is formalized under a representation map φ and instantiated in logit space (a geometric mixture of output distributions) and weight space (task arithmetic), with their computational and statistical trade-offs characterized and both variants ablated.

3. Empirical validation across domains and scales. RISE is reported to outperform RLVR-only training and on-policy self-distillation baselines on mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks, with improved sample efficiency, no additional sampling cost, and a modest 1.3–1.6× wall-time overhead.

Main Findings

• RISE beats all baselines on in-domain math averages. Both RISE variants beat every baseline on in-domain Math Avg in all three math configurations, and at least one ranks first or second on every individual in-domain benchmark. Gains are largest on competition benchmarks: on OLMo3-7B, RISE (logit) raises AIME'24 from 30.2 to 46.9 (+16.7) and Math Avg from 47.6 to 56.4 (+8.8); on Qwen3-1.7B, RISE (logit) lifts Math Avg from 45.4 to 50.2 (+4.8); on Qwen3-8B, RISE (weight) lifts Math Avg from 60.0 to 62.7 (+2.7).

• Neither extrapolation space consistently dominates, which the authors read as evidence that the gains come from the extrapolation principle itself rather than from one particular instantiation.

• Results are reproducible across seeds. Multi-seed runs give RISE (weight) Math Avg = 62.4 ± 0.2 versus GRPO 60.1 ± 0.2 on Qwen3-8B, and RISE (logit) 49.6 ± 0.4 versus GRPO 45.1 ± 0.4 on Qwen3-1.7B, across three seeds.

• Privileged-conditioning baselines give limited or negative gains. GRPO+SDPO underperforms GRPO on Qwen3-8B (55.9 vs. 60.0) and Qwen3-1.7B (43.2 vs. 45.4). SDAR and RLSD do better but stay within roughly two points of GRPO in most settings, consistent with the paper's argument that privileged-conditioning teachers are capped by in-context learning ability.

• Out-of-distribution performance is preserved. RISE maintains or slightly improves OOD accuracy: Qwen3-8B OOD Avg rises from 70.6 (GRPO) to 72.0 (RISE weight), and OLMo3-7B from 51.2 to 55.5 (RISE logit), so token-level refinement does not degrade general capabilities.

• Sample efficiency improves. RISE reaches higher evaluation accuracy in fewer steps across three model scales, with the gap widest early in training, when the extrapolated teacher supplies dense signal before RL advantage estimates stabilize.

• Multi-domain training does not dilute per-domain gains. On Qwen3-4B-Base trained on mixed math and STEM data, RISE (weight) achieves the highest Math Avg (44.8 vs. GRPO's 40.2) and STEM Avg (47.5 vs. 45.5), improving AIME'24 to 28.8 (+5.0) and TheoremQA to 51.5 (+3.7) simultaneously. GRPO+SDPO is more competitive here (STEM Avg 47.0) than in the math-only setting, yet RISE still wins without privileged information.

• Stable training on math and STEM curves. On AMC'23, both RISE variants stay above GRPO throughout training, with RISE (weight) reaching 70.0% while GRPO rises mid-training but declines to 59.4%; RISE (weight) also holds a consistent edge on SuperGPQA.

• Code-generation and agentic numerical results are not reported in the excerpt provided. The paper states that RISE outperforms RLVR-only and on-policy self-distillation across all settings including code generation and multi-turn agentic tasks, and Figure 4 is described as showing RISE converging faster than GRPO on HumanEval+ and MBPP+, but the code-generation table and the agentic task numbers fall beyond the truncated content supplied.

Methodology in Plain English

The starting observation is that a model's training run leaves a trail: each RLVR step moves the weights in some direction. If you take the vector from an earlier checkpoint (the "anchor") to the current one, that vector roughly points toward "better." RISE simply continues along that arrow, past the current point, by a factor β > 1 — producing a synthetic model that stands in for a more capable future version of the student.

Two ways of doing this arrow-continuation are offered. In weight space, you add β times the parameter difference to the anchor's parameters, exactly as in task arithmetic, and load the resulting model to read off its predictions. In logit space, you never build a model at all: you take the difference between the current and anchor checkpoints' log-probabilities at each position, scale it by β, and add it back to the anchor's log-probabilities, which amounts to a geometric mixture that amplifies the ratio between the two checkpoints' distributions.

Each training iteration then runs in two phases. First, RLVR samples rollouts from the current policy, computes outcome rewards, and applies a policy-gradient update (GRPO-style). Second, the extrapolated teacher is constructed from that update and the student is distilled toward it using a per-token divergence — the paper replaces reverse KL with Jensen–Shannon divergence for numerical stability, and approximates the full-vocabulary divergence over a top-K plus a tail bucket, with K = 100 (K = 20 for code, where the distribution is more peaked). The two phases share the same rollouts, so no extra sampling is needed.

Because a fixed β would eventually overshoot as the model approaches its optimum, β is decayed linearly from β₀ = 1.2 down to 1 across iterations. The anchor is either the previous checkpoint (η = 1) or an exponential moving average of past checkpoints (η = 0.1), which smooths the direction and enlarges the displacement; Qwen models use EMA and OLMo uses the previous checkpoint. Distillation rather than direct adoption of the extrapolated model is deliberate: the extrapolated point lies beyond the policy's trust region, so OPD acts as a trust-region projection that pulls toward the teacher's token-level distribution while the divergence term keeps the update anchored near the post-RLVR checkpoint.

The justification for extrapolating at all comes from prior analyses: post-training updates are dominated by a low-rank subspace, evolve near-linearly, and propagate to linear evolution in output log-probabilities. The authors' own runs confirm this regime — three directions capture ∼87% of the variance.

Why This Matters

Impact on research. The paper reframes self-improvement as a question about teacher construction rather than teacher replacement or noise tolerance. Prior OPD remedies — token-level gating, divergence mixing, DAgger-style rollout mixing, trajectory refinement — accept a flawed teacher and engineer around its noise. RISE argues the teacher should be the model's own extrapolated future self, and shows that a non-stationary, trajectory-derived teacher removes the static teacher ceiling that fixes the gap J(π*) − J(π_T) for external-teacher methods. It also offers a concrete alternative to privileged conditioning, whose benefits the paper measures as marginal or negative in several settings.

Real-world applications:

• Post-training reasoning models for mathematics and STEM, where RLVR provides only a scalar correctness signal and finer credit assignment is valuable.
• Code generation pipelines, where dense token-level supervision may help more than an outcome reward on syntax-heavy outputs (the paper reports evaluation on Skywork-OR1-Code but the numerical table is not in the excerpt).
• Multi-turn agentic systems (ALFWorld, WebShop), where credit must be assigned across long action sequences.
• Any setting where a stronger external teacher is unavailable, expensive, or license-restricted, since RISE needs no external model and no privileged context.

Industry relevance. The method requires no additional sampling cost, only a 1.3–1.6× wall-time overhead, and adds no external data or model into the training system — an attractive profile for teams doing RLVR post-training who want denser supervision without paying for a teacher model or a second inference stack. The weight-space variant is simple enough to implement as an adaptation of existing RLVR infrastructure.

Future Directions

• When does extrapolation break down? The analysis assumes a low-dimensional, near-linear training trajectory; the paper notes RISE does not require strict linearity but that confinement to a low-dimensional subspace limits deviation. How the method behaves for long training runs, high-curvature regimes, or architectures with different optimization geometry remains open.
• Principled scheduling of β and the anchor. The paper uses a linear β decay and ablates previous-checkpoint versus EMA anchors, but β₀ and η are treated as hyperparameters. A theory-driven schedule tied to the Proposition 3 threshold is a natural next step.
• Comparison against external-teacher OPD. A direct empirical comparison with ExOPD is described as inapplicable because ExOPD requires an external stronger model. Quantifying exactly how much of the teacher-quality gap RISE closes where a strong teacher is available would clarify its standing.
• Extension to other post-training signals. The construction converts a sparse outcome-induced parameter update into a dense token-level target. Whether the same extrapolation principle transfers to other sparse signals — preference optimization, process rewards, or tool-use feedback — is not addressed.

Target Audience

Researchers and engineers working on LLM post-training, particularly those already running RLVR or GRPO pipelines and looking for denser supervision without an external teacher. It will also interest readers following self-improvement, recursive training loops, and model-merging-adjacent techniques such as task arithmetic. The paper is advanced: readers without a background in policy-gradient methods, KL/Jensen–Shannon divergences, and low-rank training dynamics will need supporting material, though the core intuition — continue along the direction the model has been improving — is easy to state.

Authors’ abstract

On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose \textbf{RISE} (\textbf{R}ecursive \textbf{I}mprovement via \textbf{S}elf-\textbf{E}xtrapolating Policy Distillation), which constructs a synthetic teacher directly from the model's own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor---in parameter space or output logit space---RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.