Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

AI summary

Overview

Research area: Large language model post-training, specifically on-policy self-distillation (OPSD) and agentic reinforcement learning. Technical level: Advanced — assumes familiarity with KL divergence, policy-gradient methods such as GRPO, and multi-turn agent training. Scope: A controlled empirical study of how the content, format, and generation source of privileged information affect OPSD for multi-turn agents, and of whether cheap offline metrics can predict that effectiveness before training.

What This Paper Is About

OPSD lets a model improve by distilling from itself while it is given extra "privileged information" (PI) — such as a reference solution, environment observations, or a hint — that it will not have at inference time. Prior work treated the choice of PI as a fixed design decision, so it was unclear which kinds of PI actually help, whether the answer varies by model or task, and whether PI quality can be judged without paying for a full training run. This paper systematically varies PI across 16 configurations, three agentic benchmarks, and three student models, and compares three offline metrics for predicting which PI will train best.

Key Contributions

1. A systematic study of PI construction for OPSD. The authors describe it as the first controlled study of PI construction for OPSD, spanning 16 PI configurations and three student models across three multi-turn agentic benchmarks (ALFWorld, AppWorld, and τ²-retail).
2. Practical PI design recommendations. They characterize how PI content, format, generation source, and student-model choice shape OPSD gains — for example, that trajectory PI generally beats compressed hints and that skill hints are often the strongest hint format.
3. Two KL-based offline diagnostics for PI assessment and selection. They propose sampled-token reverse KL (D) and error-step KL contrast (Δd⁻_err), analyze what each measures through controlled experiments, and show that error-step KL contrast can guide per-task PI selection before costly OPSD training.
4. Combining OPSD with GRPO. They show that the most effective PI, combined with GRPO through a simple advantage-level weighting, can outperform matched GRPO baselines.

Main Findings

• Trajectory PI generally beats compressed hints. Reference-trajectory formats outperform Summary, Skill, and Checklist hints, and Instruct students benefit further when trajectories include environment observations and teacher thinking tokens.
• Richer PI can hurt already-capable models. Adding teacher thinking tokens to Action+Obs. hurts Qwen3-4B-Thinking on ALFWorld and AppWorld, and on τ²-retail adding either observations or thinking tokens causes even larger declines. The paper notes this model's pass@4 of 88.9 on τ², suggesting it already reasons effectively.
• Skills are usually the strongest hint format. Skill hints lead for all three students on ALFWorld, for both 4B students on AppWorld, and for 4B-Instruct on τ²-retail. The authors interpret this as evidence that effective PI should supply mistake analysis and reusable strategy, not just a summary of a correct solution.
• The PI generator matters most for hints. GLM-5.2 PI almost always beats Qwen3.7-Plus PI: for 4B-Instruct, GLM wins 20 of 21 comparisons, with a lead reaching 36.0 points on ALFWorld hints but staying within 3.2 points for trajectory formats.
• Self-generated hints still help. For 4B-Instruct, both self-generated Summary and Skill hints improve performance on all three benchmarks.
• PI utility is unreliable because of information leakage. Utility correlates strongly with post-training performance on AppWorld (ρ = 0.81–0.96) but is inconsistent on ALFWorld (ρ = 0.46 for 4B-Instruct, −0.56 for 4B-Thinking, −0.10 for 30B-Instruct) and on τ² (ρ from −0.06 to 0.71).
• Reverse KL tracks performance in eight of nine settings. D correlates positively with post-training performance in eight of the nine student–benchmark combinations (ρ = 0.65–0.95); the exception is 4B-Thinking on τ², where ρ = −0.84, a case where a large shift can overwrite useful existing behavior.
• Error-step KL contrast is positive in all nine settings. Δd⁻_err correlates positively with post-training performance across all nine student–benchmark combinations (ρ = 0.10–0.94), strongest for 4B-Instruct on AppWorld (ρ = 0.94) and 30B-Instruct on ALFWorld (ρ = 0.92); on τ²-retail it is ρ = 0.70 for 4B-Thinking.
• A near-perfect-utility PI can teach nothing. In the ALFWorld-Passcode variant, a "Key only" PI reaches 98.1 pass@1 with the PI in the prompt, yet training on it leaves the model exactly where it started. Both KL metrics rank it lowest among the four PIs.
• The key changes utility; the trajectory changes learning. Swapping a correct key for a decoy moves utility by 25 points but leaves training performance and both KL metrics essentially unchanged, whereas swapping the task's own trajectory for another task's trajectory nearly halves training performance to 45.0 and lowers both KL metrics.
• Error-step KL contrast improves per-task selection. Selecting each task's PI from the top 3 by Δd⁻_err beats random selection when candidates differ in source, and most when the pool mixes the student's wins and losses; among the student's winning rollouts it leads random by 5.1 points at step 40, with the gap closing by the end of the third epoch.
• Reverse KL selection helps only for a strong single-source pool. Selecting by D gives little benefit except when all candidates come from GLM-5.3-Flash; elsewhere it is slightly worse than random and fails badly when the pool mixes the student's own wins and losses.
• PI quality survives combination with GRPO. Trajectory PI remains strongest across most values of λ, followed by Skill and Checklist, and the ordering holds through λ = 0.99 on all benchmarks.
• Combined training can beat matched GRPO baselines. At λ = 0.99, Trajectory PI reaches 95.13 on ALFWorld versus a GRPO baseline of 84.96, 43.90 on AppWorld versus 42.91, and 55.04 on τ²-retail versus 48.27.
• PI mainly accelerates early learning and can overfit. On ALFWorld, PI-guided curves plateau or decline after step 100 (one epoch) while GRPO keeps improving. On AppWorld, Trajectory PI reaches nearly 100% training success at λ = 0.9 but only 39.14 test pass@1.

Methodology in Plain English

The setup is self-distillation without a separate teacher. The same model plays two roles: a student that sees only the normal prompt, and a teacher that sees the normal prompt plus privileged information. Training minimizes the reverse KL divergence from the student to the teacher's next-token distributions on trajectories the student itself generates, so the student internalizes what the PI reveals.

The authors vary the PI in several ways. Four formats keep a successful reference trajectory at different levels of detail: actions only, actions plus environment observations, actions plus reasoning, and all three. Three hint formats compress information: a short Summary, a Skill containing a success analysis, a mistake analysis, and a reusable workflow, and a Checklist built from one successful reference trajectory paired with one failed student trajectory. Hints are written by GLM-5.2, Qwen3.7-Plus, or by the student itself.

To test whether PI can be screened before training, they compute three offline metrics on training or development tasks held out from evaluation. PI utility measures how much the PI raises the model's own success rate when the PI is placed in its prompt. Sampled-token reverse KL measures how far the teacher's predictions shift from the student's on tokens the student actually sampled. Error-step KL contrast measures whether that shift is concentrated at steps a judge flags as errors, compared with task- and position-matched steps from other rollouts of the same task.

They train Qwen3-4B-Instruct-2507, Qwen3-4B-Thinking-2507, and Qwen3-30B-A3B-Instruct-2507 in SkyRL and report pass@1 from the best three-checkpoint window. The benchmarks are ALFWorld (1,600 training tasks; 140 valid_seen and 134 valid_unseen evaluation tasks, pooled), AppWorld (90 train tasks, 168 test_normal tasks), and τ²-retail (352 training tasks from the AReaL τ² dataset, 114 evaluation tasks, 40 of which are also scored on natural-language assertions judged by GPT-4.1). Two controlled studies sharpen the metric analysis: ALFWorld-Passcode, a modified ALFWorld where only a submitted key is scored, isolates leakage; and a per-task selection experiment compares training with a PI drawn from the top 3 by each metric against a random draw from the same pool, using four candidate pools (8 strong PI sources, 8 GLM-5.3-Flash winning trajectories, 8 self wins, and 8 self wins plus 8 self losses). Finally, GRPO and OPSD are combined at the advantage level with a weight λ, where λ = 0 is vanilla OPSD and λ = 1 is GRPO.

Why This Matters

Impact on research. The paper reframes PI selection as a measurable design variable rather than a fixed choice, and it shows that the most intuitive proxy — does the PI make the model succeed when it can read it? — is the wrong one. Because leaked answers can inflate or deflate that proxy arbitrarily, the KL-based diagnostics offer a more principled screening criterion, and the error-step contrast in particular connects PI quality to whether supervision lands on actual mistakes.

Real-world applications:

• Training customer-service and tool-use agents, where PI can be built from successful reference trajectories, environment observations, or rubrics.
• Building coding and computer-use agents from long-horizon interaction logs, the setting the authors cite for methods like Composer 2.5.
• Reducing the cost of post-training pipelines by screening dozens of candidate PI formats offline instead of running a full distillation job for each.
• Diagnosing why a distillation run failed — for example, a large distributional shift that overwrites already-correct behavior in a reasoning model.

Industry relevance. OPSD removes the need for a separate teacher model, which makes it attractive for teams without a larger model to distill from. The paper's practical guidance — use error-step KL contrast to select PI when candidates vary, pair strong PI with a strong GRPO signal, and reduce the PI weight once its benefit plateaus — maps directly onto production post-training schedules.

Future Directions

• Construct PI that targets the trained model's remaining errors, rather than repeatedly training on the same static PI, so supervision stays relevant as the policy improves.
• Provide multiple diverse PI instances per task, which the authors suggest may deliver a similar benefit to refreshing the PI.
• Design adaptive PI schedules that reduce or reshape the distillation signal once its marginal benefit decays, avoiding the plateau and overfitting seen after the first epoch.
• Make the thinking-model case work. Thinking models benefit less consistently and can be harmed by rich trajectory PI; the authors connect this to privileged distillation penalizing productive reconsideration tokens, leaving open how to supervise reasoning models without disrupting useful behavior.

Target Audience

Researchers and engineers working on LLM post-training, agentic reinforcement learning, and distillation — particularly those who already run GRPO-style pipelines and are considering adding self-distillation. It is most useful to readers comfortable with KL-based objectives and multi-turn agent evaluation, and least suited to beginners seeking an introduction to distillation.

Authors’ abstract

On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL ($D$) and error-step KL contrast ($Δd^{-}_{\mathrm{err}}$). The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Empirically, selecting each task's PI by $Δd^{-}_{\mathrm{err}}$ also improves post-training performance over random selection, most when the PI candidates differ widely. Finally, effective PI remains beneficial when combined with GRPO, allowing the combined objective to outperform a strong GRPO baseline. Overall, our results provide practical guidance for constructing and evaluating PI in OPSD training.