Continually Post-training Small Models via Synthetic Self-distillation

AI summary

Overview

• Research area: Continual post-training of large language models (continual learning, supervised fine-tuning, knowledge distillation), with a specific focus on small language models.
• Technical level: Advanced. The paper formalizes aligner selection with KL-divergence objectives and describes full fine-tuning configurations, though its core ideas are explained with concrete analogies and experiments.
• Scope: The paper diagnoses why self-distillation fine-tuning fails at small model scales and proposes SynthSelf, an off-policy synthetic self-distillation method that uses a minimally larger, same-family aligner to generate in-distribution training data.

What This Paper Is About

Continually post-training a language model is a two-sided problem: the model must acquire new knowledge while retaining what it already knows. Self-Distillation Fine-Tuning (SDFT) handles this by having the model rewrite its own demonstrations in-context, but that requires strong in-context learning and instruction-following ability, which small models lack. The paper's goal is to make continual post-training work at small scales (0.5B, 1.5B, 3B) by outsourcing the in-distribution rewrite to a slightly larger model from the same family.

Key Contributions

1. A diagnosis of why continual learning fails. The authors show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift. In Experiment 1, forgetting magnitude across five non-math benchmarks correlates with policy distribution shift at Spearman rho = -0.85.
2. Evidence that SDFT's self-supervision degrades with scale. An evaluation of SDFT's self-generated offline dataset shows that smaller models yield weaker supervision: average accuracy falls to 60% and CoT traces are dropped 8.2% of the time, regardless of prompt.
3. SynthSelf, a new method. The method imitates SDFT's in-distribution rewrite by outsourcing it to a minimally larger, same-family aligner, and couples this with off-policy supervision and rejection sampling to give small models high-quality supervision without breaking their original distribution.
4. An aligner-selection criterion. The paper formalizes choosing an aligner that minimizes distributional gap to the base model subject to two constraints: its rewrites must be closer in distribution than gold SFT data, and its supervision quality must exceed the base model's own.

Main Findings

• SynthSelf leads acquisition across all four settings. Averaged over all settings (science, science mixture, tool-use, medicine), SynthSelf with a one-size-larger aligner reaches 56.4 acquisition versus 52.2 for SFT, 44.1 for SDFT, 39.8 for DFT, and 46.1 for ASFT, while matching the strongest baselines on 14 retention tasks.
• Gains are not just from using a bigger model. Controlling for teacher model size by comparing against off-policy and on-policy distillation with the identical aligner, SynthSelf improves acquisition by +5.7%p to +16.9%p averaged over four settings and three base scales.
• In-distribution matters more than aligner strength. Stronger out-of-distribution aligners (Qwen3.5-27B, Gemma-4-31B) generate higher data-generation accuracy but introduce severe distributional shift. Averaged over the 0.5B, 1.5B, and 3B base models, more in-distribution aligners outperform out-of-distribution ones by +2.5 and +4.6 percentage points in acquisition on science and tool-use, with lower NLL.
• Cost savings. Using a same-family aligner reduces data generation inference costs by 62.9% to 76.9% in FLOPs, and the paper reports a 69.9% reduction in alignment computational cost (FLOPs) relative to using a larger, stronger out-of-distribution teacher.
• Rejection sampling helps. Ablating the offline data generation and filtering step shows it consistently improves both SynthSelf and off-policy distillation, which the authors attribute to small models being highly sensitive to noisy supervision.
• Prompt differences do not explain the gains. Against four SDFT prompt variants including one matched to SynthSelf, the method outperforms prompt-matched SDFT by +18.96%p and +1.68%p on acquisition and retention respectively.
• Fewer but better examples. Rejection sampling shrinks the training dataset (e.g., by 17.1% for tool use), yet SynthSelf improves acquisition by +4.2%p on average over SFT.
• Medical domain is an exception. SynthSelf does not significantly improve over off-/on-policy distillation there, which the authors attribute to the aligner models already having perfect knowledge of the training data, making demonstration injection largely redundant. DFT's medical performance deteriorates especially significantly.
• Distribution shift experiments. Experiment 1 trained on 23 diverse math datasets size-matched to 2.5M completion tokens; Experiment 2 held the corpus fixed (OpenR1-Math-220k) and varied only target shift across high, mid, and low shift variants of 2M completion tokens each, showing monotonic improvement in target-domain performance and retention as shift decreases.

Methodology in Plain English

The authors start by testing a hypothesis: that training on targets far away from the model's native distribution is what causes both bad learning and forgetting. They fine-tune Qwen2.5-3B-Instruct on many math datasets and measure how far the resulting model drifts from the original, then check whether drift predicts forgetting. They repeat this within a single corpus, rewriting the same data in three progressively less shifted forms (full reasoning trace, final solution only, and the model's own verified-correct solution).

Having shown that staying in-distribution is what matters, they ask why SDFT cannot achieve it at small scales. They generate an offline dataset from the model's own rewrites and measure accuracy and CoT retention, finding both degrade as the model shrinks.

SynthSelf then keeps SDFT's principle — keep the data in the model's own distribution — but changes the source. Instead of asking the small model to rewrite its own demonstrations in-context, a slightly larger model from the same family does the rewriting. Because this generation happens offline rather than during training, the authors can reject any rewrite whose extracted answer does not match the gold answer, keeping only same-answer, differently-phrased examples. The small model is then trained with standard supervised fine-tuning on this filtered, in-distribution dataset. The paper provides a qualitative example where the gold and rewritten reasoning are paraphrases of each other but both end in answer A.

Why This Matters

Continual post-training has mostly been demonstrated on large models, while the models people actually deploy on-device are small. This work identifies a concrete, measurable failure mode (distribution shift) and shows that a lightweight, same-family aligner can fix it without a larger, stronger, out-of-distribution teacher — which is both cheaper and more effective in these experiments.

Real-world applications:

• On-device assistants. Small models that need to pick up new domain knowledge without losing general instruction-following, commonsense, or coding ability.
• Latency-specialized and draft models. Small models used as speculative decoders or verifiers, cited by the authors as a key motivation for small-model continual learning.
• Domain customization. Adapting a small model to science, tool-use, or medical data where the paper's acquisition benchmarks are drawn from.
• Cost-constrained deployment pipelines. Where a cloud teacher is either too expensive or forbidden by data-residency and privacy constraints, a same-family aligner within the same model family avoids sending data to a third-party model.

Industry relevance: the paper reports FLOPs savings (62.9% to 76.9% for data generation, 69.9% for alignment) and a training dataset reduced by as much as 17.1% in one setting, both of which translate directly into lower post-training budgets.

Future Directions

• Removing the same-family requirement. The method's main limitation is needing a larger, same-family aligner. The authors suggest adapting out-of-family aligners by adding the base model's answer in-context with an appropriate prompt to generate an in-distribution rewrite.
• Scaling coverage supervision. The authors note practitioners can scale k in pass@k, trading inference compute for larger supervision coverage.
• Extending beyond the evaluated scales and domains. The study covers 0.5B, 1.5B, and 3B Qwen2.5-Instruct models on science, tool-use, medicine, and a four-subject science mixture; larger and more diverse settings are untested here.
• Understanding domain-specific ceilings. The medical domain showed no significant gain over off-/on-policy distillation because the aligner already knew the data perfectly; when aligner knowledge makes rewriting redundant remains an open question.

Target Audience

Researchers and engineers working on continual learning, knowledge distillation, and post-training of language models. It is most directly useful to practitioners who fine-tune small models under compute or privacy constraints and need to avoid catastrophic forgetting, and to researchers interested in the alignment between training-data distribution and model retention. Readers should be comfortable with KL divergence, on-policy versus off-policy training, and supervised fine-tuning terminology.

Authors’ abstract

Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities. Consequently, SDFT fails on smaller models critical for low-latency, on-device use. To enable continual learning at smaller scales, we first identify the root cause of failure. We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning. We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner. Coupled with off-policy supervision and rejection sampling, this provides the high-quality supervision small models require without breaking their original distribution. Evaluations spanning 4 acquisition datasets (including a mixture) and 14 retention benchmarks show SynthSelf consistently outperforms 6 methods (including off- and on-policy distillation) across 3 small model scales. Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.