Ask to Distill: Learning to Synthesize Data for On-Policy Distillation

AI summary

Overview

Research area: Large language model post-training, specifically on-policy distillation (OPD) and synthetic training-data generation, with reinforcement learning used to train a data-generating policy.

Technical level: Advanced. The paper assumes familiarity with KL divergence, top-K candidate distributions, PPO/GRPO-style policy optimization, and on-policy distillation.

Scope: The paper proposes Ask-to-Distill (A2D), a framework that trains a question-generating model to produce prompts whose student-generated trajectories expose the most useful teacher supervision for a specific frozen student–teacher pair, then uses those prompts for standard OPD.

What This Paper Is About

On-policy distillation trains a small student model on its own sampled trajectories, using a stronger teacher's token-level distributions as supervision. Because OPD is surprisingly prompt-efficient, the training questions themselves matter a great deal: a question determines which states the student visits and therefore where teacher feedback becomes available. The problem is that a newly generated question has no reference answer, so there is no obvious way to tell whether it is a good distillation question. A2D answers this by scoring a question not by its correctness or difficulty, but by how much usable teacher–student distributional signal it elicits along student responses, and then learning to generate more such questions.

Key Contributions

1. A reframing of synthetic prompts as distribution elicitors. The paper formulates synthetic prompt construction as optimizing the distributional supervision a question exposes for a particular student–teacher pair, rather than as supplying target answers for imitation.

2. A question-level reward built from support-aware token signals. The reward multiplies teacher–student discrepancy (a teacher-to-student KL over the union of the two models' top-K candidate sets) by compatibility (the teacher's probability mass on the student's top-K candidates), and adds sublinear length normalization, a response-budget penalty, and a batch-diversity penalty. It requires no reference answers, no task-specific correctness reward, and no student update per candidate question.

3. A complete, self-contained pipeline. A synthesizer initialized from the student is trained with GRPO against a frozen student–teacher pair; generated questions are filtered and deduplicated (token-set Jaccard plus semantic deduplication with a frozen Qwen3-Embedding-4B) into a prompt-only bank; standard candidate-restricted OPD is then run on that bank.

4. Validation across model scales and three domains. Experiments cover mathematical reasoning (three student–teacher pairs), code generation (three pairs), and agentic tasks (one pair), with downstream comparisons, reward-component ablations, an initialization ablation, an iterative-training study, and learning-signal analyses.

Main Findings

• A2D has the highest average score among OPD methods in every reported student–teacher setting. Across three math pairs, three code pairs, and one agentic pair, A2D's average exceeds real-bank OPD and the other synthetic-data baselines.

• Mathematical reasoning, Qwen3-1.7B student / Qwen3-4B-Non-Thinking-RL-Math teacher: A2D reaches an average of 46.2 (teacher-gap recovery 90.2%), versus 41.1 for Student-Generated OPD, 44.1 for KL-Filtered OPD, 43.0 for Vanilla OPD, 42.1 for Evol-Instruct, and 42.3 for R-Zero. Student starts at 25.7, teacher at 48.4.

• Mathematical reasoning, DeepSeek-R1-Distill-Qwen-1.5B / JustRL-DeepSeek-1.5B: A2D averages 51.6 (recovery 99.9%), versus 48.2 student-generated, 48.6 KL-filtered, 50.5 vanilla, 49.8 Evol-Instruct, and 50.0 R-Zero. Student 38.8, teacher 51.7.

• Mathematical reasoning, DeepSeek-R1-Distill-Qwen-7B / Skywork-OR1-Math-7B: A2D averages 64.5 (recovery 81.3%), versus 60.8 student-generated and 62.9 vanilla. Student 58.1, teacher 66.0.

• Code generation, Qwen3-4B / Qwen3-4B-Instruct-2507: A2D averages 39.6 (recovery 82.9%), versus 33.5 student-generated, 35.0 KL-filtered, 34.5 vanilla, 34.0 Evol-Instruct, and 34.9 SQLM. Student 28.7, teacher 41.8.

• Code generation, Qwen3-4B / Qwen3-30B-A3B-Instruct-2507: A2D averages 35.7 (recovery 32.8%), versus 29.5 student-generated and 32.2 KL-filtered. Student 28.7, teacher 50.0.

• Code generation, Qwen2.5-Coder-3B-Instruct / Qwen2.5-Coder-32B-Instruct: A2D averages 21.2 (recovery 12.3%), versus 19.8 student-generated and 20.6 SQLM. Student 19.7, teacher 32.0. This is the smallest margin reported.

• Agentic tasks (τ²-Bench), Qwen3-4B / Qwen3-4B-Thinking-2507: A2D averages 41.9 (recovery 95.2%), versus 37.3 student-generated, 40.9 KL-filtered, and 40.4 vanilla. Student 14.4, teacher 43.3.

• Optimized questions are shorter and denser. The paper reports that generated questions yield 20% shorter responses and 28% more total signal than DAPO-17k-Math, and that optimized questions elicit denser supervision and more total signal from shorter student responses.

• Every reward component matters. Removing compatibility, the training-time diversity term, or the explicit length penalty reduces accuracy on all three of AIME24, AIME25, and AIME26.

• KL filtering alone helps but is not sufficient. Selecting the top 20% of real questions by teacher–student KL improves over student-generated prompts in most settings but remains below A2D.

• Iterating the pipeline gives only small additional gains. On the code pair with Qwen3-4B and Qwen3-30B-A3B-Instruct-2507, round 1 averages 40.5 (recovery 43.3%), round 2 reaches 40.8 (+0.3), and the final round R=3 reaches 41.0 (+0.5, recovery 44.8), against a student of 29.0 and teacher of 55.6.

• Synthesizer initialization: RL improves both student-initialized and teacher-initialized synthesizers. Student initialization with RL is competitive across benchmarks and exceeds the teacher reference on LCB, while remaining below it on APPS-Intro and APPS-Interview.

Methodology in Plain English

The pipeline has three stages, each run once.

1. Train the question generator. The generator starts as a copy of the student. For each generation instruction, it samples a group of candidate questions. The frozen student answers each candidate (one response per question), and the frozen teacher is evaluated on those same student-written prefixes. At each token position, the method computes two things: how much the teacher and student disagree over their combined top-K candidate tokens, and how much teacher probability mass sits inside the student's own top-K candidates. Multiplying these (with the disagreement capped at a calibrated reference scale) gives a per-position score, so a position is only valuable if the teacher both disagrees and cares about tokens the student could actually learn. Scores are summed over the response and divided by length raised to a power between 0 and 1, then multiplied by a penalty that grows with response length and a penalty for questions that land in large text-similarity clusters within the batch. This composite reward is attached to the final token of the generated question, and GRPO updates only the generator.

2. Curate the questions. The trained generator is sampled at scale. Malformed outputs, repeated n-grams, low-diversity text, unbalanced math delimiters, answer leakage, missing context, and references to absent figures are filtered out. Remaining questions are deduplicated first by token-set Jaccard similarity, then greedily by embedding cosine similarity using a frozen Qwen3-Embedding-4B, which is used only for this step and never for reward or training. The output is a bank of questions with no answers.

3. Run ordinary on-policy distillation. The original student is trained on responses it samples itself, with the frozen teacher supervising only the student's top-K candidate tokens at each position. The update is a PPO-style clipped objective with a probability-weighted, reverse-KL-style advantage, one inner epoch per rollout batch, and no value function or KL anchor to a reference model. The downstream procedure is held fixed across all prompt-source comparisons, so differences come from the questions alone.

No external training questions, labeled examples, or auxiliary models are used for synthesizer training. Calibration of the discrepancy scale uses questions sampled from the initial student policy only.

Why This Matters

The paper shifts attention in distillation-data construction from "what does this question ask" to "what supervision does this question extract from this particular student–teacher pair." That matters because OPD is known to be highly prompt-efficient, so the choice of a small number of questions can carry much of the training signal. It also shows that disagreement alone is a poor filter: large teacher–student divergence can sit outside the student's reach and therefore be unusable, which is exactly the gap A2D's compatibility term targets.

Real-world applications:

• Compact math tutoring and reasoning models: a 1.7B student recovers 90.2% of the gap to its teacher on competition math benchmarks in this study.
• Code assistants built on smaller or specialized models: improvements appear across LiveCodeBench, MBPP+, APPS subsets, and CodeContests for both general and code-specialized families.
• Agentic tool-use systems: the τ²-Bench results cover airline, retail, and telecom domains, where a 4B student reaches 95.2% teacher-gap recovery.
• Cheap model compression without answer labels: because no reference answers or verifiers are needed, the pipeline can be applied where ground truth is expensive or unavailable.

Industry relevance: the method is self-contained (no external question corpus, no auxiliary reward model, no per-candidate student training), operates on the prompt-construction axis, and is complementary to token-selection, trajectory-reweighting, or loss-modification techniques, making it practical to layer onto an existing OPD training stack.

Future Directions

• Scaling the evaluated regime. The largest student reported is DeepSeek-R1-Distill-Qwen-7B and the largest teacher Qwen3-30B-A3B-Instruct-2507; whether the reward remains discriminative at larger scales and with smaller teacher gaps is untested.

• Improving the surrogate's fidelity. The authors explicitly describe the reward as a surrogate for learning value rather than an estimate of downstream improvement or gradient magnitude, leaving open how tightly it correlates with actual performance gain.

• Making iteration pay off. Three rounds of re-running the pipeline on the updated student yielded only +0.5 average over round 1, so how to schedule or reuse the synthesizer across rounds remains an open question.

• Combining with orthogonal methods. The paper positions A2D as complementary to supervision-position selection, trajectory reweighting, and modifications of the downstream distillation objective, but does not test such combinations.

• Broadening domain coverage. Agentic evaluation uses a single student–teacher pair, and the code-benchmark section is truncated in the supplied content, so per-benchmark code details beyond the reported averages are not fully specified here.

Target Audience

Machine learning researchers working on distillation, synthetic data, and RL-based post-training; engineers building post-training pipelines who already have a teacher–student pair and want better prompts without labeled data; and readers interested in self-play and self-evolving data-generation methods who want a distributional alternative to task-success-based rewards. Beginners will find the high-level idea accessible, but the method section presupposes comfort with KL divergences, top-K truncation, and policy-gradient objectives.

Authors’ abstract

On-policy distillation (OPD) can succeed with surprisingly few training questions, motivating a view of questions as carriers of distributional knowledge transfer. How can we assess whether a newly generated question is valuable for distillation, and learn to generate better questions---without access to its correct answer? We introduce Ask-to-Distill (\method), a framework that treats synthetic questions as distribution elicitors and optimizes the teacher supervision they expose along student-generated trajectories. A student-initialized synthesizer is optimized using a distributional surrogate for question value, computed from a frozen student--teacher pair without reference answers, correctness rewards, or per-candidate student updates. The reward combines teacher--student disagreement with teacher probability mass on the student's candidate support, together with length and diversity controls. The generated questions are filtered and deduplicated into a prompt-only bank for standard OPD. Across mathematical reasoning, code generation, and agentic tasks, our method improves average performance and teacher-gap recovery over unoptimized student-generated prompts and KL-filtered real questions, while matching or exceeding real-bank OPD in the evaluated settings. Signal analyses further show that optimized questions can elicit denser supervision and more total signal from shorter student responses, supporting elicited distributional supervision as an objective for synthetic distillation data. Our code, data and models are publicly available at https://github.com/GasolSun36/Ask_to_Distill.