A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning

AI summary

Overview

Research area: Large language model reasoning, reinforcement learning with verifiable rewards (RLVR), on-policy self-distillation, and token-level credit assignment in post-training.

Technical level: Advanced. The paper assumes familiarity with group-relative policy optimization, KL divergence between teacher and student distributions, and per-token log-ratio objectives.

Scope: The paper tests whether a reference-conditioned "scoring" teacher provides usable error-level credit, shows that it does not, and proposes replacing it with a generated critique that is then used to condition the same scoring pass.

What This Paper Is About

On-policy self-distillation tries to turn the single bit of feedback that RLVR gives per rollout (correct or incorrect) into dense, token-level supervision: the same model scores a student's own rollout while the reference solution sits in its context. The authors ask whether that scoring signal actually contains corrective credit — whether it points at the tokens where the reasoning went wrong and pushes probability away from them. They find it does not, argue that locating an error is itself a search problem the scoring pass is never asked to solve, and show that letting the same model generate a critique first recovers the missing signal.

Key Contributions

1. A direct empirical test of the privileged scoring signal. Using gold error spans annotated in student rollouts, the authors measure both readouts consumed by self-distillation objectives — the full-distribution KL and the token log-ratio — and show that neither localizes errors nor suppresses erroneous student tokens.

2. Evidence that the failure is structural, not informational. The same model, with the same reference solution, localizes errors at a far higher rate when it is allowed to generate a critique instead of only scoring. This separates "what information is available" from "what computation is performed."

3. Critique-conditioned self-distillation. A method that first generates a critique of the current rollout (a verbatim quote of an erroneous sentence plus an explanation), then conditions the scoring teacher on that critique, producing a dense signal that localizes errors and is corrective within error spans.

4. Initial evidence that the signal is usable at inference and in training. The critique is used both to resume wrong rollouts at test time and as an advantage signal in cold-start GRPO, including on all-wrong groups where verifiable rewards alone provide no gradient.

Main Findings

• The reference-conditioned signal is not localized at errors. On Knights & Knaves (K&K), error and clean-sentence traces are nearly indistinguishable for both the localization and correction signals. On DeepMath the reference induces only a small error-specific shift, most visible near sentence boundaries, and much weaker than under critique conditioning.

• The correction signal often fails or points the wrong way. Across three students scoring their own rollouts, reference-conditioned error-span Z scores stay close to the rollout's own clean baseline, and no reference setting satisfies the corrective criterion. On DeepMath L7-10, the Qwen3-1.7B teacher assigns more probability to the student's erroneous token when conditioned on the reference (correction +0.17, 95% CI excluding 0).

• A bigger scoring teacher does not help. Re-scoring the same Qwen rollouts with an 8B teacher leaves reference-conditioned scores similarly small, while critique-conditioned signals become stronger.

• The signal cannot recover the verifier's one bit. Neither reference-conditioned signal reliably ranks correct rollouts above incorrect siblings of the same problem: all twelve within-problem AUC confidence intervals include chance. Pooled deviations that do move away from chance also appear under content-free controls, and a trained probe over richer trace-level summaries does no better than those controls.

• Generating finds the errors that scoring cannot. With gpt-5.2 as an oracle critic, a quote lands within five sentences of a gold error on 57% of K&K rollouts, 62% of L5-7 and 67% of L7-10. Qwen3-8B reaches 26/22/27 and Qwen3-1.7B critiquing itself reaches 20/18/21. Practical failures are paraphrasing instead of quoting and format violations; quote-match is 98/96/95 for the oracle.

• Critique conditioning restores localization and correction. Oracle critiques produce positive localization and negative correction on every task and every student, at several times the magnitude of the reference-conditioned signal. Self-critiques preserve the same pattern at reduced magnitude, and the effect strengthens with model scale.

• The released probability mass goes to self-correction language. Under critique conditioning, the share of post-error tokens whose argmax becomes a self-correction marker such as "Wait", "But" or "However" rises to 23.9 against a 7.7 control on K&K, 9.0 against 5.2 on L5-7, and 10.6 against 4.8 on L7-10. Under reference conditioning, error windows show no such excess (for example 4.9 against 4.9 on K&K).

• Quote carries localization, reason carries correction. Ablating the critique shows that removing the reason leaves localization largely unchanged, while removing the quote weakens it. Removing the reason shrinks the corrective shift substantially and, with the 8B teacher, can reverse it (quote-only correction of +0.21 on L5-7 and +0.03 on L7-10).

• The signal works at inference. On 55 wrong Olmo-3-32B-Think rollouts from 39 L7-10 problems, 19 of which the model never solves in eight samples, resuming from the critique's flagged sentence with the critique item and the cue yields 9.3% sample accuracy on the unsolved problems versus 4.3% for a fixed-depth cut with the cue alone (13.4% versus 10.9% on all problems).

• The signal works in training. On four K&K puzzles Olmo-3-7B-Think never solves by sampling, the critique-conditioned arm solves two puzzles that the GRPO control never solves. On a

Authors’ abstract

On-policy self-distillation promises to replace the sparse outcome signal of verifiable-reward RL with dense token-level credit: a teacher conditioned on the reference solution scores the student's own rollout, and the resulting distributional shift is distilled back into the policy. We investigate whether this scoring signal actually contains corrective credit. Using gold error spans in student rollouts, we test the two signals consumed by self-distillation objectives, the full-distribution KL and the token log-ratio, for two properties: whether they localize errors and whether they push probability away from erroneous student tokens. They exhibit neither of these properties nor can they reliably distinguish correct from incorrect rollouts within the same problem. We argue that this failure is structural: locating an error in a reasoning trace is itself a search problem, while a scoring teacher is never explicitly asked to perform that search. The same model, with the same privileged information, localizes errors substantially better when allowed to generate a critique. This forms the motivation for \emph{critique-conditioned self-distillation}: first generate a critique of the current rollout, then condition the scoring teacher on that critique. The resulting signal localizes errors, is corrective within error spans, and shifts probability toward tokens that initiate backtracking such as ``Wait'', ``But'', and ``Actually''. We further show that the critique is useful both at test time, where resuming a wrong rollout with the critique in context converts math problems the model cannot solve by sampling, and in cold-start GRPO, where it provides an advantage signal on prompts for which verifiable rewards alone provide none.