VASPO: Self-Distillation with Privileged Value Estimation

AI summary

Overview

Research area: Natural Language Processing, specifically reinforcement learning for LLM reasoning (RLVR), on-policy self-distillation, and value-based credit assignment.

Technical level: Advanced. The paper assumes familiarity with PPO, GAE, GRPO, KL divergence, and actor–critic training.

Scope: The paper proposes VASPO, a PPO-based training framework that gives a critic and a self-teacher the same privileged reference solution so that the critic can estimate continuation returns and the teacher can modulate token-level advantages.

What This Paper Is About

Outcome-only reinforcement learning gives a reasoning model a single reward at the end of a response, but does not say which tokens earned it. On-policy self-distillation adds a denser signal by letting a "self-teacher" that has seen a worked solution score the student's tokens, but that signal reflects local token preferences rather than whether the student can actually finish the problem on its own.

VASPO's goal is to combine both signals inside PPO: a critic that also sees the reference solution predicts the expected return of the student's unguided continuation, while the self-teacher only rescales selected positive advantages within a bounded range in order to avoid overriding the outcome-grounded credit.

Key Contributions

1. Positional characterization of privileged information. The authors measure how the two signals behave across response positions: the critic's prediction benefit grows toward the end of a response, while self-teacher divergence is high near the beginning, falls through the middle, and rebounds at the final position bin.
2. The VASPO framework. A self-teacher and a critic read the same fixed privileged information at identical student-generated prefixes. The critic produces GAE advantages from predicted continuation returns, and the teacher modulates selected positive advantages through clipped probability ratios, while the actor is trained and deployed on the standard prompt only.
3. Empirical gains on math and logic. Across six mathematical benchmarks, VASPO improves mean accuracy over PPO by 2.59 percentage points on Qwen3-4B variants and 1.81 percentage points on OLMo3-7B variants, and it achieves the highest mean accuracy among evaluated methods on Knights and Knaves with Qwen3.
4. Controlled ablations. The paper isolates the privileged critic, the self-teacher modulation, the token-level mask, the privileged-information content (solution plus answer versus answer only), the critic pre-update, and advantage whitening.

Main Findings

• Mathematical reasoning gains: VASPO reaches a mean of 53.17% on Qwen3-4B-Instruct-2507 versus 50.58% for PPO, 51.59% for DAPO, 50.88% for RLSD, 49.76% for Vanilla, and 34.80% for OPSD. On OLMo3-7B-Instruct-DPO it reaches 33.42% versus 31.61% for PPO, 32.47% for RLSD, 32.26% for DAPO, 29.81% for Vanilla, and 23.67% for OPSD.
• Per-benchmark leadership on Qwen3: AIME24 56.87, AIME25 43.96, AIME26 51.15, HMMT25 29.17, Minerva 64.75, OlympiadBench 73.12 — the highest score on each of the six benchmarks in the reported table.
• Per-benchmark leadership on OLMo3: AIME24 29.58, AIME25 24.58, AIME26 24.90, HMMT25 14.90, Minerva 47.75, OlympiadBench 58.83 — again the highest on each benchmark.
• Logical reasoning: On Knights and Knaves with Qwen3, VASPO has the best average (60.81) against RLSD (60.34), DAPO (59.88), PPO (59.19), and Vanilla (58.30). It leads the 4–8-role split (75.31), the 10-role split (58.06), and the 11-role split (40.62), but RLSD edges it out on 9-role puzzles (70.19 versus 69.25). All methods degrade as puzzle size grows.
• Raw privileged information is not enough: OPSD's mean collapses to 34.80 on Qwen3 math and 23.67 on OLMo3 math, which the authors read as evidence that direct access to solutions is insufficient without outcome-grounded credit.
• The privileged critic is the largest single component: Removing it (leaving the OPSD objective) drops the math average to 35.59 and the KK average to 23.30. Removing self-teacher modulation instead gives 55.74 and 59.65; removing both gives 53.50 and 59.19.
• Selective modulation matters: Dropping the token-level mask lowers accuracy to 54.95 on math and 59.60 on KK, degrading every benchmark and role group in both domains.
• Full solutions beat answers alone: Replacing solution-plus-answer with answer-only input lowers accuracy to 54.59 on math and 58.91 on KK.
• Critic pre-update matters: Removing the pre-update gives 55.00 on math and 58.84 on KK.
• Advantage whitening hurts: Applying global whitening after teacher fusion reduces accuracy to 54.41 on math and 58.24 on KK, because subtracting a batch-wide mean can flip the sign of fused advantages.
• Moderate fusion is best: The default clip width of 0.2 yields higher final AIME accuracy than either disabling teacher modulation (0) or widening it to 0.5.
• Value prediction versus calibration: On fixed continuations from a frozen Qwen3 actor, the solution-conditioned critic has the highest explained variance and lowest Brier error across the three checkpoints, but the baseline critic retains the lowest expected calibration error.
• Where each signal helps: Solution conditioning degrades accuracy at early prefixes and improves it at late prefixes, with the largest late-stage gains on incorrect source responses; the pattern holds for medium and long responses under length controls, though the late gain remains uncertain in the long-response group. Training rewards are similar across methods, but VASPO holds higher explained variance and better held-out accuracy.

Methodology in Plain English

The student model sees only the original problem and generates a response. Before that rollout, the researchers fix one piece of privileged information — the reference solution and answer — and give the same text to two helper models at every prefix the student produces.

The first helper is a critic. It reads the prefix plus the solution and predicts how likely the student is to eventually get the problem right if it continues on its own without the solution. The critic is refreshed on each new batch of rollouts before any advantages are computed, and because the reward is binary and arrives only at the end (with discount and GAE lambda set to 1), the advantage for a token simplifies to the verifier's outcome minus the critic's predicted value at that prefix. This is what gives token-level credit that varies by position.

The second helper is a self-teacher, an exponential-moving-average copy of the policy conditioned on the solution. It scores the same sampled tokens the student produced. The log-probability gap between teacher and student becomes a weight, clipped to a narrow band around 1 (clip width 0.2). That weight is applied only to tokens with a positive advantage and a teacher–student gap large enough to pass a tanh-based threshold (tau and delta both 0.02). The fused advantage is the original advantage times one plus the mask times the weight minus one, so the sign never changes and negative advantages are untouched. The actor then takes a standard clipped PPO update with these fixed advantages, with no subsequent whitening.

Training used the verl codebase with full-parameter AdamW, actor and critic learning rates of 1e-6 and 5e-6, global batch 128, mini-batch 32, eight responses per prompt, PPO clip 0.2, sampling temperature 1.0 during training and 0.6 during evaluation, top-p 0.95, top-k 20, and a maximum response length of 8,192 tokens. For Qwen3-4B, student rollout averaged 191.97 seconds per step over 200 training steps.

Why This Matters

Impact on research. The paper reframes privileged information as useful for two distinct jobs rather than one. Prior combinations such as RLSD, SSPO, and SRPO use privileged information only in a teacher branch on top of group-relative advantages; VASPO puts it into a learned PPO critic so outcome-grounded advantages can vary by prefix before any teacher modulation. The positional analysis also gives a concrete reason for the split: teacher divergence and critic benefit peak in different parts of a response. This offers a diagnostic template — measure signal profiles by position before deciding how to fuse them.

Real-world applications:

• Post-training math and science reasoning tutors where verified answers and worked solutions already exist in the training data.
• Code generation pipelines where unit tests supply the verifier and reference patches supply the privileged context.
• Data-efficient fine-tuning of small open models (4B–7B) that cannot afford a separate larger teacher model, since the self-teacher is a copy of the policy itself.
• Logic and constraint puzzles, where the paper shows transfer from 4–8-role training puzzles to out-of-distribution 9-, 10-, and 11-role puzzles.

Industry relevance. The method is implemented in verl, uses standard PPO infrastructure, and reuses dataset-provided solutions rather than requiring a stronger external model. For teams already running PPO or GRPO on reasoning tasks, the changes are localized to the critic's input and an advantage rescaling step. The authors also flag the cost side: conditioning both readers on full solutions expands the context window for teacher and critic, increasing training cost.

Future Directions

1. Multi-turn dialogue. VASPO is designed for single-turn generation; conversational settings where history and available evidence evolve across turns remain open.
2. Interactive and agentic tasks. Under the current protocol, environmental feedback obtained at the end of a response cannot be used retroactively as privileged context for earlier prefixes of that same trajectory, so the timing rules for when information becomes admissible need to be worked out.
3. Evidence quality. Reference solutions can contain flawed intermediate reasoning even when the final answer verifies, which injects noise into both readers; filtering or validating intermediate steps is an open problem.
4. Cost and calibration. Reducing the extra context cost of solution-conditioned readers, and addressing the calibration trade-off (higher explained variance and lower Brier error but worse ECE than a baseline critic) are natural follow-ups.

Target Audience

Researchers and graduate students working on LLM post-training, reinforcement learning with verifiable rewards, and distillation for reasoning models. It is most useful to practitioners who already understand PPO and GAE and who want a concrete recipe for injecting training-only reference solutions into a critic. Readers looking for an introduction to RL for LLMs should start elsewhere, since the paper assumes the PPO/GRPO/OPSD background and moves quickly into advantage-level detail.

Authors’ abstract

On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at https://github.com/cii030/VASPO.