Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

AI summary

Overview

Research area: Post-training of large language models, specifically on-policy distillation (OPD) and reinforcement learning with verifiable rewards (RLVR), with an emphasis on asynchronous training infrastructure efficiency.

Technical level: Intermediate to Advanced. The paper assumes familiarity with reverse KL distillation, PPO-style clipped policy gradients, and GRPO; the core idea is simple, but the notation and system discussion are dense.

Scope: The paper introduces Teacher-Gated On-Policy Distillation (TGOPD), a method that verifies a teacher model's per-prompt reliability with a small set of verifier-scored teacher probes before admitting its dense token-level supervision, and it evaluates this method on 4B and 35B students across mathematics, code, and instruction following. (Paper: arXiv:2609.02998v2, AllSpark Team, public technical report dated August 28, 2026.)

What This Paper Is About

On-policy distillation trains a student model using dense, token-by-token guidance from a stronger frozen teacher, but it applies that guidance uniformly to every prompt without checking whether the teacher is actually correct on that prompt. Because reverse KL is mode-seeking, a teacher that is confidently wrong can push the student hard in the wrong direction. TGOPD solves this by spending the teacher's otherwise-idle GPU time generating a few probe answers per prompt, scoring them with the same verifier used for the student, and using that pass rate as a gate: dense distillation if the teacher passes, verifier-grounded GRPO if it fails.

Key Contributions

1. A prompt-level reliability gate. The paper defines teacher reliability as $R_T(x) = \mathbb{E}_{y \sim \pi_T(\cdot \mid x)}[r(x,y)]$ and estimates it online with $K_T$ verifier-scored teacher probes, $q_T(x) = K_T^{-1}\sum_k r(x, \hat{y}^k)$, which is unbiased for any probe budget with variance $R_T(x)(1-R_T(x))/K_T$. A hard gate $g(x) = \mathbb{1}[q_T(x) \geq \tau]$ converts this into a binary admission decision.

2. Mutually exclusive supervision routing. The per-token advantage is $\hat{A}^{\mathrm{TGOPD}}{i,t} = g(x)\hat{A}^{\mathrm{OPD}}{i,t} + (1-g(x))\hat{A}^{\mathrm{GRPO}}_{i,t}$, which the paper emphasizes is a selector rather than an interpolation: OPD and GRPO never jointly supervise the same prompt. Vanilla OPD and pure GRPO are the two degenerate endpoints of this rule.

3. A diagnostic showing confidence is not reliability. Using ten offline teacher responses per prompt across 2,400 prompts per domain, teacher self-confidence separates high- from low-reliability prompts at AUROC 0.51 on code and 0.73 on math; in the low-reliability regime the teacher's highest-confidence sampled answer is still wrong 84% of the time on code and 61% on math.

4. A system realization that reclaims idle teacher capacity. Probes run concurrently with student generation, raising measured teacher-node GPU utilization from 9.8% to 78.9% in the 4B single-domain run while adding only a modest measured end-to-end cost.

Main Findings

• TGOPD leads the seven-benchmark average at both scales. In single-domain training it scores 56.67 at 4B and 63.53 at 35B, exceeding Vanilla OPD by 2.10 and 1.09 points respectively, and the strongest baseline by average (TrOPD) by 0.16 and 0.87 points. It achieves the best distillation score on nine of the 14 benchmark–scale pairs, including ties, and surpasses the corresponding domain teacher on seven.

• Gains over Vanilla OPD hold in all six domain–scale settings. On the within-domain average, the 4B improvements are 2.47 points in math, 2.00 in code, and 1.65 in instruction following; at 35B they are 0.67, 2.25, and 0.55. Math gains most at 4B and code at 35B.

• The code results are the clearest case for gating. At 35B, Vanilla OPD scores 60.2 on LiveCodeBench, improving only 0.2 points over the base model's 60.0, while TrOPD, RG-OPD, and RLSD-style fall below the base model by 2.2, 3.0, and 2.3 points. TGOPD reaches 64.0, gaining 4.0 points over base and 3.8 over Vanilla OPD, and exceeding the code teacher by 1.3 points on LiveCodeBench and 1.1 points on OJBench (tying TrOPD at 28.7). At 4B, TGOPD raises LiveCodeBench from 45.0 to 47.1, closing roughly 55% of the base-to-teacher gap versus 40% for Vanilla OPD.

• Some benchmark-specific regressions exist. At 35B, TGOPD scores 43.7 on IFBench, slightly below Vanilla OPD's 44.2, while improving IFEval from 89.9 to 91.5. TrOPD remains strongest on 4B HMMT-Feb and LiveCodeBench; TGOPD ties TrOPD on 35B OJBench and ties Vanilla OPD on 35B HMMT-Feb.

• The RLSD-style variant shows the cost of using teacher information everywhere. Applying teacher-scaled magnitude with verifier-determined direction on every prompt falls below the base model on nine of the 14 benchmark–scale pairs, including a 5.5-point drop on 4B IFEval and a 2.3-point drop on 35B LiveCodeBench.

• Multi-domain OPD improves without modification. Training one student on math, code, and IF simultaneously with domain-routed teachers, at 199 training steps with $K_T = 3$ and $\tau = 2/3$, the 4B seven-benchmark average rises from 53.40 to 54.54 (+1.14) and the 35B average from 60.99 to 61.94 (+0.95). The largest single-benchmark gains are on AIME 2026 (+3.85 at 4B), IFBench (+2.94 at 4B), and OJBench (+0.87 at 4B, +4.31 at 35B). Regressions (LCB at 4B −0.48; AIME 2025 −0.15 and IFBench −0.41 at 35B) are all under half a point.

• Teacher idleness is structural. Without probes, the teacher node averages between 5.0% and 9.8% utilization across four configurations, with 57–78% of all 15-second samples below 5%. Multi-domain OPD is the more wasteful mode (78% idle at 4B, 70% at 35B) versus 57–59% under single-domain OPD, because three domain-specialist engines share one node and each runs only on its own domain's prompts.

• Probing raises utilization sharply. Teacher-node mean utilization goes from 9.8% to 78.9% (4B SOPD), 8.8% to 82.8% (35B SOPD), 7.0% to 66.6% (4B MOPD), and 5.0% to 57.7% (35B MOPD); the fraction of samples below 5% falls to 0–2%. Cluster-wide utilization improves by +18.0 and +11.8 points under SOPD and +12.7 and +8.6 under MOPD (4B, 35B). Student-rollout utilization changes by between −1.8 and +2.1 percentage points. In a matched 35B CodeIO comparison, mean step time increases by 5.9% over 500 aligned rollout cycles.

• The gate threshold has a broad optimum. A sweep at $K_T = 5$ on 4B math over 99 steps shows an inverted-U: at $\tau = 3/5$ all three benchmarks peak (AIME 2026 at 73.1, AIME 2025 at 64.4, HMMT-Feb at 58.0, exceeding Vanilla OPD by +1.9, +3.3, and +3.7), while at $\tau = 5/5$ HMMT-Feb falls below Vanilla OPD (52.0 vs 54.3). The peak at a 60% pass rate is close to the default $\tau = 2/3$.

Methodology in Plain English

The starting point is a standard asynchronous distillation loop: a student generates a group of rollouts for a prompt, and a stronger frozen teacher scores every token of those rollouts to produce a dense per-token learning signal. The teacher's scoring pass is cheap and cannot start until the student finishes, so the teacher's GPUs sit mostly idle.

TGOPD fills that idle window with a small audit. While the student is still decoding, the teacher generates $K_T$ probe answers for the same prompt, and the same verifier that already scores student rollouts (normalized answer matching for CodeI/O output prediction, execution-based verification for CodeI/O input prediction, and rule-based verification for math and IF) scores the probes. The fraction that passes is $q_T(x)$. If $q_T(x) \geq \tau$, the prompt gets dense OPD supervision; otherwise the teacher's signal is withheld and the update uses verifier-grounded GRPO with group-mean-centered rewards. Both candidate advantages are computed eagerly for pipeline regularity, but the binary gate selects exactly one, and teacher scoring still runs unconditionally on every prompt because making it conditional would add a data-dependent branch for little savings.

The main experiments compare against Vanilla OPD, TrOPD, RG-OPD, and an RLSD-style variant, all sharing the same student initialization, frozen domain teacher, corpus, and pipeline. Two student scales are used: a dense 4B model (Qwen3.5-4B) and a mixture-of-experts 35B model (Qwen3.6-35B-A3B with 3B active parameters). Each domain teacher is produced by applying GRPO to the same base model. Prompts come from DAPO-Math-17K (math), the input–output prediction questions of CodeI/O (code), and filtered Nemotron-Cascade 2 (IF). Evaluation uses AIME 2025 and AIME 2026 (64-run accuracy average), HMMT-Feb 2025 (32-run average), the code-generation subtask of LiveCodeBench (pass@1, 6-run average), OJBench overall (C++/Python aggregate), IFBench, and IFEval. Training runs on the slime framework, with IcePop applied to all compared methods to stabilize train–inference mismatch from asynchronous rollout. Default settings are $K_T = 3$ and $\tau = 2/3$, a two-of-three majority.

Why This Matters

Research impact. The paper reframes teacher selection in distillation as an admission-control problem rather than a weighting problem. Prior reliability-aware methods—EOPD (forward KL at high-entropy teacher tokens), TrOPD (token-level trust regions from decoding agreement), REOPOLD (clipped token rewards from likelihood ratios and student entropy), RG-OPD, RLSD, and SPOT—use distributional evidence such as entropy, agreement, or likelihood gaps, or they regulate individual trajectories or token branches. TGOPD instead aggregates repeated verifier-scored teacher outcomes into a single prompt-level decision made before any dense supervision is applied. The reported code-domain results, where confidence separates reliability at only AUROC 0.51, give concrete evidence that outcome-based verification carries information that distributional proxies miss.

Real-world applications:

• Post-training pipelines for reasoning and coding assistants, where a domain-specialized teacher is already deployed and its output correctness can be checked automatically.
• Cost-constrained training clusters, where reclaiming idle teacher accelerators (9.8% to 78.9% utilization) reduces wasted reserved capacity.
• Multi-domain model training, where one student learns from several routed specialist teachers and each prompt must be matched to a teacher that is actually reliable on it.
• Safety and quality control in distilled models, since withholding supervision on prompts where the teacher fails limits the propagation of confidently wrong behavior into the student.

Industry relevance. Asynchronous distillation is already standard in large-scale post-training, and the paper's measurements show that teacher nodes are idle by construction, not by misconfiguration—57–78% of 15-second samples sit below 5% utilization. A method that converts that idle capacity into a better training signal, at a measured 5.9% increase in mean step time in the matched 35B comparison, is directly actionable for teams that already run verifier-scored RL. The requirement is modest: a verifier that the pipeline already needs for GRPO, plus a configurable probe budget.

Future Directions

1. Probe budget and threshold beyond the tested settings. The paper empirically uses $K_T = 3$ and $\tau = 2/3$ in main experiments and performs a finer sweep only at $K_T = 5$ on 4B math; whether these choices transfer to other domains, scales, and verifiers is not established.

2. Normalized versus un-normalized GRPO fallback. The implementation omits the standard-deviation normalization of GRPO, and the authors explicitly state this is an implementation convention rather than a requirement of the gate, leaving a controlled comparison with normalized variants to future work.

3. Reducing the residual overhead. The overlap between probes and student generation is described as substantial but not perfect, and the matched 35B CodeIO comparison shows a 5.9% increase in mean step time; shrinking that remainder is an open engineering question.

4. Whether reliability estimation can be made cheaper or reused. TGOPD spends fresh teacher probes per prompt; the paper does not report whether reliability estimates could be cached, amortized across steps, or predicted from a cheaper signal once an outcome-verified set of prompts has been observed.

Target Audience

This paper is most useful to machine learning engineers and researchers who run post-training pipelines for large language models, particularly those working with on-policy distillation, GRPO-style verifiable-reward RL, or multi-teacher domain routing. It also suits infrastructure and systems engineers focused on GPU utilization and asynchronous training throughput, since roughly half the paper is a measurement study of where teacher compute goes. Readers without prior exposure to reverse KL distillation or PPO-clipped surrogates will need background reading, but the gate itself—probe the teacher, verify, admit or withhold—is easy to grasp conceptually.

Authors’ abstract

On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.