On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

AI summary

Overview

Research area: Natural Language Processing — post-training of large language models, specifically on-policy self-distillation, self-training, and synthetic data generation, extended to vision-language models.

Technical level: Advanced. The paper assumes familiarity with supervised finetuning, reinforcement learning with verifiable rewards (RLVR), on-policy distillation, KL/Jensen–Shannon divergences, and EMA teacher–student setups.

Scope: The paper proposes On-Policy Self-Training (OPST), a method that lets a model generate its own questions and privileged reasoning traces from nothing but coarse topic names, filter them intrinsically, and train on them via on-policy self-distillation — evaluated on Gemma 4 E2B/E4B across MMLU, GPQA, GSM8K, HellaSwag, and VQAv2/POPE.

What This Paper Is About

Post-training methods for LLMs keep relocating human supervision rather than removing it: SFT needs reference solutions, RLVR needs gold answers, and on-policy self-distillation (OPSD) typically needs a privileged context such as the ground-truth solution. Curating such data is expensive — the paper cites roughly 800,000 human step labels for a single mathematics corpus (Lightman et al., 2024) and open instruction datasets built by over 13,000 volunteers (Köpf et al., 2023). OPST asks whether a model can improve using only coarse topic descriptors like "Abstract Algebra," generating both the problems and the privileged information used to train on them, with no human-curated questions or answers.

Key Contributions

1. A data-free post-training formulation. The paper defines a setting where training is seeded only by a set of coarse topic descriptors, from which the model synthesizes a training set of generated queries paired with privileged information (a rationale plus a candidate answer). Neither a curated prompt distribution nor ground-truth answers are available.

2. Topic-seeded curriculum generation with two intrinsic filters. A generator produces questions and privileged information from a topic; pseudo-verification checks that the answer is reproducible in a fresh context and under a paraphrase, while a consensus gate discards samples the current student already answers consistently across rollouts, so the curriculum's difficulty evolves with the student.

3. Training on synthetic privileged information as context rather than as target. Instead of imitating the generated answer (SFT) or rewarding agreement with it (GRPO), OPST feeds it to an EMA teacher as privileged context, giving dense token-level supervision on the student's own rollouts using a skewed Jensen–Shannon divergence with α = 0.8.

4. Extension to multimodal self-training (Visual OPST). With unlabeled images from OpenImages V4 and visual topic descriptors, the model generates image-grounded questions plus privileged scene captions and/or bounding-box-grounded spatial relations, verified across augmented views.

Main Findings

• Large in-domain gains on Gemma 4 E2B without thinking. MMLU rises from 37.5 to 47.7 (+10.2), GPQA from 28.8 to 32.8 (+4.0), HellaSwag from 48.7 to 56.2 (+7.5), and the four-benchmark average from 38.3 to 45.6 (+7.3). GSM8K is not reported for the non-thinking setting.

• Gains persist with thinking enabled. On Gemma 4 E2B, GPQA rises from 27.8 to 34.3 (+6.5) and GSM8K from 72.7 to 80.9 (+8.2), while MMLU gains are smaller (+1.2, from 69.9 to 71.1). On Gemma 4 E4B with thinking, MMLU rises from 76.0 to 80.1 (+4.1), GPQA from 43.9 to 48.0 (+4.1), GSM8K from 84.4 to 93.0 (+8.6), and the average from 71.3 to 76.0 (+4.7).

• Dense context beats dense targets and sparse rewards on identical data. OPST exceeds SFT on the same self-generated data by 5.6 / 1.3 points (2B / 4B) without thinking and by 2.7 / 3.8 points with thinking. The only setting where SFT is ahead is 4B HellaSwag without thinking, by 0.4 points.

• No curated problem set is needed to beat methods that use one. Among methods without reference solutions, OPST has the best average in all four settings, exceeding u-OPSD by 1.0 to 2.8 points and R-Zero by 1.4 to 5.2 points. At 2B without thinking it recovers 88% of the supervised OPSD gain (+7.3 of +8.3), and with thinking it matches supervised OPSD at both model sizes (+0.2 and +0.7 points).

• Multi-domain training works with limited interference. Training on all three in-domain topic sets improves the in-domain average by 7.6 points and comes within 1.0 point of single-domain training. Out-of-domain retention is mixed: the OOD average improves by 0.8 points, but MATH-500 regresses by 4.9 points. Adding GSM8K topics to MMLU gives a larger MMLU gain (+11.3) than MMLU topics alone (+10.2) and the best OOD average (+3.9).

• Some gains are not domain-specific. Training on HellaSwag and GSM8K topics, with no MMLU topics at all, still improves MMLU by 5.1 points.

• Visual OPST works with unlabeled images only. On Gemma 4 E2B, combining scene captions and spatial relations gives the largest VQAv2 gain (+6.9, from 49.1 to 56.0); spatial relations alone give +6.6 and captions alone +4.9. POPE recall improves by 4.5 points at 2B and 6.2 points at 4B. At 4B, spatial QA alone is best on both VQAv2 (+3.7) and POPE, so the two privileged-information forms are not consistently complementary.

• Both filters matter. Without filtering, OPST gains +5.2 over base on MMLU (42.7); pseudo-verification adds +2.7 (45.4) and the consensus gate a further +2.3 (47.7).

• The privileged information is doing the work. Removing it shrinks the MMLU gain from 10.2 to 3.1 points; a placebo from a different problem on the same topic yields only 2.7. The rationale alone recovers 8.3 points and the answer alone 7.1.

• Robustness to wrong generated answers. With filters disabled and 75% of candidate answers replaced by wrong ones, OPST loses only 1.9 points and stays 3.3 points above base, whereas SFT loses 5.7 points and falls 4.4 points below base.

• The EMA teacher is important and is the model evaluated. With β = 1.00 (no EMA), MMLU is 38.1; β = 0.20 gives 44.2, β = 0.05 gives 47.7, and β = 0.01 gives 45.2. The paper notes that averaging the student's weights yields a more accurate model than the student itself.

Methodology in Plain English

The method is a loop with four moving parts.

1. Generate a curriculum from a topic name. Given a broad topic such as "Abstract Algebra" or "Object Counting," the model is prompted to write a problem and, alongside it, the "privileged information": a rationale and a candidate answer. These questions can be multiple-choice or open-ended and are sampled at a temperature to encourage diversity. The generator is either the EMA teacher (so data quality can improve over rounds) or a frozen copy of the initial model.

2. Throw away bad and useless problems. Two filters run on every generated sample. Pseudo-verification asks the model to solve its own question in a fresh context without seeing the answer it authored, then asks it to paraphrase the question and solve that too; if the answer changes, the sample is discarded as ambiguous. The consensus gate then has the student attempt the question several times; if it agrees with the candidate answer on every rollout, the problem is too easy and is dropped. Because this gate is re-evaluated against the current student, the retained problems stay near the student's capability frontier as it improves.

3. Train by matching a better-informed version of itself. The student generates a reasoning rollout on the question. A teacher — an exponential moving average of the student's weights — sees the same question plus the generated privileged information, and produces a next-token distribution at each step of the student's own rollout. The student is trained to match that distribution using a skewed Jensen–Shannon divergence with α = 0.8, which is bounded and favors the mode-seeking regime, so localized artifacts in a bad synthetic hint cannot dominate the gradient. Crucially, the generated answer is a hint the teacher reasons from, not a target to copy, so the teacher can fill in steps the rationale omits and can resist errors the model finds implausible.

4. Repeat. Generation, verification, and training are repeated over multiple rounds with duplicate problems removed; student and teacher weights carry over, so a stronger teacher produces better problems, better targets, and better verification in the next round.

For vision, the generator additionally receives an unlabeled image and a visual topic, and the privileged information becomes a detailed scene caption, bounding-box-grounded spatial relations, or both. Visual verification solves the question on several augmented views of the image (color jitter, rescaling, random cropping — but not horizontal flips, which would flip left/right answers) and requires the answer to agree.

Why This Matters

Impact on research. The paper reframes on-policy self-distillation as a self-training framework and shows that the last piece of external supervision — the problem set itself — can be removed. It provides evidence that dense, token-level teacher context extracted from synthetic privileged information is a better use of self-generated data than treating it as an SFT target or an RL reward, and it quantifies how much of the gain comes from each filter, from the rationale versus the answer, and from the EMA teacher.

Real-world applications:

• Domain adaptation without annotation — adapting a model to a specialized field (legal, medical, internal codebases) given only category labels or topic names rather than curated question–answer sets.
• Low-resource languages and niches — improving models where no crowdsourced instruction data or reference solutions exist.
• Multimodal assistants — bootstrapping visual question answering and reducing object hallucination from unlabeled image collections.
• Continual deployment updates — repeatedly refreshing a deployed model on its own generated curriculum without a fresh human annotation cycle each time.

Industry relevance. The work was done at Google with Gemma 4 models, and the practical claim is economic: the paper cites ~800,000 human step labels for one math corpus and 13,000+ volunteers behind open instruction datasets, and OPST's data-free configuration lands within reach of fully supervised OPSD on average (and matches it in the thinking settings at both model sizes).

Future Directions

• Correctness-aware verification. The authors note that pseudo-verification measures self-consistency, not correctness, so consistent errors still enter training; they propose tool- or retrieval-grounded verification as a remedy.
• Label-free stopping criteria. With no held-out labeled validation set, it is unclear when to stop iterating or which round to keep; the paper flags this as an open direction.
• Escaping the knowledge ceiling. OPST "can only amplify existing knowledge," so combining it with grounding mechanisms that can introduce genuinely new information remains open.
• Scaling and automation of the seed set. The topic descriptors are hand-chosen and experiments are limited to small models (Gemma 4 E2B and E4B); whether the loop holds for larger models and automatically derived topic descriptors is untested here.

Target Audience

Researchers and engineers working on LLM post-training, self-distillation, and synthetic data — particularly those who already understand SFT, RLVR, and distillation objectives and want to know whether the human-curated problem set can be eliminated. It is also relevant to practitioners building domain-adaptation or multimodal pipelines who have topic labels and unlabeled data but no annotated question–answer pairs, and to readers interested in how much of a model's improvement comes from the form of supervision rather than from additional information. Beginners will find the high-level idea accessible but the method section demanding.

Authors’ abstract

On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.