Skip to content
AI.info

Advanced techniques

Knowledge Distillation and Teacher–Student Learning

Learn how a student model can imitate a stronger teacher through soft targets, intermediate features, and carefully chosen transfer data.

By the end you can

Analogy

The apprentice who studies the expert's reasoning patterns

The recipe card supplies the dish name and nothing else. Weeks beside the chef reveal close flavor substitutes, nearly acceptable mistakes, and subtle distinctions. Soft teacher outputs provide a similar signal by exposing relative preferences among outputs rather than only the winner.

An apprentice eventually asks why. A student model never does, and it copies useful structure and systematic bias alike, so real labels and independent evaluation must keep the imitation honest.

The apprenticeship also never quite finishes. Self-distillation makes that visible: a ResNet-20 on CIFAR-100 taught by a copy of itself, a student with exactly the teacher's capacity learning from exactly that teacher. After 300 epochs the two agreed on 78.95% of the training set. Not 100%. And not because the student was too small to hold what the teacher knew.

Distillation is guided imitation: the teacher supplies richer behavior, while real labels and evaluation keep the student honest.

Why softened predictions can teach more than one-hot labels

A hard class label says which answer should win. A teacher probability distribution can also say that a wolf is more similar to a dog than to an airplane. These relative probabilities — sometimes called dark knowledge — provide gradients about the structure among alternatives. Temperature scaling softens the teacher logits so non-winning classes receive more visible probability mass.

The 2015 paper that made the case put the whole intervention into a single clause. Its small MNIST net was “regularized solely by adding the additional task of matching the soft targets produced by the large net at a temperature of 20”. Same architecture, same data, one extra task. Hinton and two colleagues published it in March 2015, listed as “NIPS 2014 Deep Learning Workshop”.

The student is commonly trained with a mixture of ordinary supervised loss and a distillation loss that matches teacher outputs. The balance matters. Too much teacher imitation can preserve teacher mistakes or override reliable labels. Too little can reduce distillation to ordinary training.

Distillation is not limited to final outputs. A student can match intermediate features, attention maps, pairwise relationships, or teacher-generated data. Each choice imposes compatibility assumptions between teacher and student architectures. And each choice is worth points, as the next comparison shows with three students of identical size.

The teacher offers a structured target distribution; the student still needs an independent definition of correctness.

Case

One student that recovered most of a ten-model ensemble

On MNIST a large regularized net made 67 test errors. A smaller net — two hidden layers of 800 rectified linear units, no regularization — made 146. Give that same small net one extra job, matching the large net's soft targets at a temperature of 20, and it made 74. Section 3 of the 2015 paper reports all three counts in one sentence: “This net achieved 67 test errors whereas a smaller net with two hidden layers of 800 rectified linear hidden units and no regularization achieved 146 errors … it achieved 74 test errors”. The 146 and the 74 are the same 800-unit net. The soft targets are the only difference between them.

The speech experiment is a three-row table, on a system trained on “about 2000 hours of spoken English data”. Baseline: 58.9% frame accuracy, 10.9% word error rate. Ten-model ensemble: 61.1% and 10.7%. Distilled single model: 60.8% and 10.7%. One student recovered most of what ten models had gained, and gave up nothing on word error rate.

The figures survive an independent copy. Hinton hosts the same paper at the University of Toronto. Its section 3 carries the identical sentence and the identical 67, 146 and 74 error counts, its Table 1 the same six figures.

Comparison

What part of the teacher should the student imitate?

Different transfer targets trade architectural freedom for a richer signal, and at a fixed budget the trade has been measured. Microsoft Research set three students side by side in the MiniLM paper: each 6 layers, 768 hidden, 66M parameters, each distilled from the same 109M BERT-base teacher. The score is an average over SQuAD 2.0 F1 plus seven GLUE dev tasks (MNLI-m, SST-2, QNLI, CoLA, RTE, MRPC, QQP).

DistilBERT matches soft target probabilities plus a cosine embedding loss on embedding outputs. It averages 77.6. TinyBERT matches embedding outputs, hidden states and self-attention distributions, layer to layer. It averages 79.1. MiniLM matches last-layer attention distributions plus value–value relations. It averages 80.4. The teacher averages 81.5. Same parameter count, same teacher, 2.8 points between the weakest and the strongest choice of what to imitate.

Of its own student, MiniLM's abstract claims: “In particular, it retains more than 99% accuracy on SQuAD 2.0 and several GLUE benchmark tasks using 50% of the Transformer parameters and computations of the teacher model.”

Two cautions travel with the table. Those are the NeurIPS 2020 camera-ready figures, and the arXiv preprint of the same paper lists DistilBERT at 75.2, not 77.6; the version you cite changes the spread. And a third organisation ranks the same families in the same order on the GLUE test set — Huawei's Noah's Ark Lab reports BERT-base at 79.5, DistilBERT6 at 76.8, TinyBERT6 at 79.4. Hugging Face states DistilBERT's own headline separately: a 40% size reduction, 97% retention, a 60% speed-up.

FigureComparison · 3 columns

Response distillation

Match teacher logits, probabilities, or generated outputs.

  • Architecture agnostic in many settings
  • Simple to add to supervised training
  • Can copy calibration defects
  • Classic soft-target distillation

Feature distillation

Align selected hidden representations or attention maps.

  • Transfers internal structure
  • Needs dimension or layer matching
  • Can constrain the student too strongly
  • Useful when architectures are related

Relation distillation

Preserve distances, similarities, or interactions among examples.

  • Focuses on representation geometry
  • Allows more architectural freedom
  • Requires careful batch construction
  • Useful for retrieval and metric learning

Visual

The teacher–student training pipeline

The same transfer set can carry ground-truth labels, teacher outputs, or both. At production scale it is a concrete artefact you can count. DeepSeek built its open-weight students by having “directly fine-tuned open-source models like Qwen and Llama using the 800k samples curated with DeepSeek-R1”. The peer-reviewed report of that release appeared in Nature in September 2025, where the authors write: “To enable broader access to powerful AI at a lower energy cost, we have distilled several smaller models and made them publicly available.”

The last step is the one with a price attached. DeepSeek reported DeepSeek-R1-Distill-Qwen-32B at 72.6 pass@1 on AIME 2024, against 50.0 for QwQ-32B-Preview. Hugging Face's Open R1 project re-ran the same released checkpoints on its own LightEval harness. It measured 69.7 on AIME 2024, 95.6 on MATH-500 where DeepSeek reported 94.3, and 63.1 on GPQA Diamond where DeepSeek reported 62.1. Identical weights, a different harness, three benchmarks that moved in two different directions. Evaluating independently is not a formality that confirms the vendor's number. It is a second measurement that disagrees with it.

FigureProcess · 5 steps
  1. 1

    Prepare the teacher

    Freeze or version a model whose behavior is worth transferring.

  2. 2

    Choose transfer data

    Use labeled, unlabeled, synthetic, or production-like examples.

  3. 3

    Generate teacher targets

    Store logits, probabilities, features, or sequences with provenance.

  4. 4

    Train the student

    Blend teacher matching with ordinary supervised objectives.

  5. 5

    Evaluate independently

    Measure student quality and failure modes without treating teacher agreement as truth.

Same checkpoints, two harnesses: 72.6 and 69.7 pass@1 on AIME 2024.

Example

Where distillation creates practical value

Compression is common, but not the only reason to distill, and each of these has a published instance behind it.

  • Edge vision: Transfer a server-grade image model into a student that meets mobile memory and latency limits — Google Research distilled BiT-ResNet-152x2 into a ResNet-50 at 82.8% top-1 on ImageNet.
  • Speech recognition: Compress an ensemble or large acoustic model into a single production model — the 2015 distilled single model reached 60.8% frame accuracy against the ten-model ensemble's 61.1%, both at 10.7% word error rate.
  • Search ranking: Use a slow cross-encoder teacher to train a faster bi-encoder or compact reranker.
  • Specialist transfer: Distill several experts or a routed system into one simpler student for a defined task set.
  • Data expansion: Label a large unlabeled pool with a teacher, then train a student with strong filtering and confidence controls — DeepSeek's 800,000 R1-generated samples are this recipe at scale.
  • Self-distillation: Use a model or later checkpoint as its own teacher — the Born-Again Networks paper says “rather than compressing models, we train students parameterized identically to their teachers”, and its DenseNet-based students beat those teachers at 3.5% error on CIFAR-10 and 15.5% on CIFAR-100 (ICML 2018). A later study qualifies the mechanism: “it is becoming well-known that in self-distillation the student fails to match the teacher and, paradoxically, student generalization improves as a result”. The gain comes from imperfect imitation, not from better imitation.

Key idea

A smaller model can become a cleaner copy of the wrong behavior

Teacher outputs encode the teacher's blind spots, class imbalance, calibration, and policy decisions. A student can reproduce those errors even when its aggregate accuracy is close. Where a compressed model's errors actually land has been measured.

A ResNet-18 trained on CelebA to predict the Blond attribute was pruned to 95% sparsity. The overall false-positive rate moved 12.72%. For the Male subgroup it rose 49.54%; for everyone else, 6.32%. The split tracks representation rather than difficulty: Blond-and-Male is 0.85% of the training set, against 14% for Blond-and-not-Male. Hooker and colleagues at Google Research name the pattern bluntly: “Compression cannibalizes performance on low-frequency attributes in order to preserve overall performance.” Their paper introduces Compression Identified Exemplars, the examples on which the compressed and uncompressed models part company, precisely because the aggregate number will not show them to you.

An independent survey repeats the finding with its own citation: “pruning a model trained to identify people with blond hair to 95% sparsity increased the average false-positive rate for men by 49.54%, but by only 6.32% for others”. That is five researchers at ETH Zürich and IST Austria, writing in the Journal of Machine Learning Research in 2021.

So compare student and teacher on protected slices, rare labels, uncertainty, and out-of-distribution examples. Include ground-truth labels where available. Consider down-weighting or rejecting low-confidence teacher outputs. An aggregate that moves 12.72% can be hiding a 49.54%.

An overall 12.72% move in false positives was 49.54% for a 0.85% subgroup and 6.32% for everyone else.

Steps

A distillation experiment that separates compression from imitation quality

Run the study with a fixed deployment budget and a fixed evaluation suite. Step 2 is the one teams skip, and Huawei's Noah's Ark Lab priced it exactly. Before distilling anything, the team pre-trained a label-only twin of its own student: “BERTTINY means directly pretraining a small BERT, which has the same model architecture as TinyBERT4.”

Both models are 4 layers, d=312, d_i=1200, 14.5M parameters, 1.2B FLOPs, a 9.4x inference speed-up on a single NVIDIA K80. Both were submitted to the official GLUE evaluation server. The label-only twin averaged 70.2 on the GLUE test set. The distilled twin averaged 77.0, against the 109M BERT-base teacher's 79.5. In the paper's words, “TinyBERT4 is consistently better than BERTTINY on all the GLUE tasks and obtains a large improvement of 6.8% on average”. Identical size, identical latency, 6.8 points apart. Read the gap carefully: TinyBERT4's edge combines the teacher signal with the paper's task-specific data augmentation, so 6.8 prices the whole recipe, not soft targets alone.

The baseline can also win, which is why it is not decoration. At the 66M budget in MiniLM's table, the non-distilled BERT_SMALL baseline — plain MLM pre-training, no teacher — averages 79.1, above distilled DistilBERT's 77.6, while truncated BERT-base falls to 76.2. Without step 2 there is no way to tell which of those three stories your student is in.

FigureProcess · 6 steps

  1. 1. Define the student budget

    Set latency, memory, throughput, and energy targets before architecture search.

  2. 2. Train a label-only student

    Establish what the compact architecture can learn without a teacher.

  3. 3. Add response distillation

    Sweep temperature and teacher-loss weight.

  4. 4. Test richer targets only if needed

    Add features or relations through controlled ablations.

  5. 5. Inspect teacher–student disagreements

    Separate teacher mistakes, student capacity limits, and transfer-data gaps.

  6. 6. Select on the Pareto frontier

    Choose the quality–latency tradeoff that fits the product, not the largest student.

Example

Keep a teacher–student disagreement notebook

Disagreements reveal whether the student lacks capacity, the transfer data lacks coverage, or the teacher itself is unreliable. The honest prior is that the student will not match. A 2021 study from NYU and Google Research opens with exactly that: “We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood: there often remains a surprisingly large discrepancy between the predictive distributions of the teacher and the student, even in cases when the student has the capacity to perfectly match the teacher.” Their measured case is a ResNet-20 on CIFAR-100, BatchNorm replaced by LayerNorm, self-distilled. The student is the teacher's twin. It still does not agree with it.

  • Teacher correct, student wrong: Inspect capacity, temperature, or intermediate supervision — but do not assume capacity is the culprit: “with SGD we achieve 83.3% agreement when training for 5k epochs compared to 78.95% when training for 300 epochs” is a same-capacity student, and the missing agreement is an optimization failure.
  • Teacher wrong, student copies: Preserve independent labels so teacher bias is visible rather than rewarded; agreement with a teacher that is wrong scores as success on every teacher-matching metric.
  • Teacher uncertain, student overconfident: Evaluate calibration and the effect of hard-label mixing, since the softened distribution is the part of the teacher you chose to transfer.
  • Teacher and student disagree off-domain: Check whether the transfer corpus represents deployment conditions — in the same paper's ResNet-56 CIFAR-100 self-distillation under Combined Augs, “the agreement drops even further, to just 60% in self-distillation!”
  • Student improves on labels: Treat the gain as evidence that imitation should not be the only objective — the Born-Again Networks beat their own teachers at 3.5% CIFAR-10 error and 15.5% CIFAR-100 error while agreeing with them less.
  • Latency target missed: Revisit architecture, sequence length, batching, or quantization instead of increasing distillation loss complexity; TinyBERT4's 9.4x speed-up on a single NVIDIA K80 came from 14.5M parameters and 1.2B FLOPs, not from a richer loss.

Position

Distillation recovered most of the ensemble, and “most” is where the decision lives

Nothing in this lesson's case says the student caught the teacher. It says the student came close, twice, and both times the remainder is on the page. On MNIST the large regularized net made 67 test errors and the 800-unit net trained on labels alone made 146. Adding the task of matching the large net's soft targets at a temperature of 20 brought that same small net to 74. Seventy-four is a long way from 146. It is also seven errors worse than the teacher.

The speech numbers are more exact about the trade. Against a 58.9% baseline, the ensemble of ten models reached 61.1% frame accuracy and the distilled single model reached 60.8%. The ensemble gained 2.2 points; the student kept 1.9 of them. On word error rate, the figure nearer to what a user experiences, ensemble and student both landed at 10.7% against the baseline's 10.9%. That is a strong result and a bounded one: one speech task, one MNIST experiment, both from the same 2015 paper, and one model running where ten had run before.

The remainder is not a peculiarity of 2015 or of small students. The self-distillation study found the same shortfall where capacity cannot explain it — 78.95% train agreement after 300 epochs and 83.3% after 5,000, student and teacher identical. Google Research paid the other side of that bill, distilling BiT-ResNet-152x2 into a ResNet-50: “with a total number of 9600 epochs for distillation, we set the new ResNet-50 SOTA 82.8% on ImageNet”. Its Figure 4 adds that the teacher “is always reached eventually, after a much larger number of epochs than one would ever use in a supervised training setup”. Fidelity is purchasable. The currency is epochs, and 9,600 of them is the quoted price.

Distillation is sold as compression that costs nothing. The evidence supports something more useful than that: a trade whose size was measured. It is also why the experiment in this lesson trains a label-only student before it brings in the teacher. The 146 is what makes the 74 mean anything, just as BERT_TINY's 70.2 is what makes TinyBERT4's 77.0 mean anything. Without that baseline, a distilled student is a model with a score. Even a student matching its teacher's aggregate exactly would still owe the calibration, rare-label and off-domain comparisons this lesson asks for — a 12.72% overall move can be 49.54% on one subgroup. What crosses from teacher to student is behavior, and the blind spots travel in the same signal as the skill.

The student finished 0.3 points of frame accuracy short of the ensemble, and that gap is what gets priced.

Figure

Distillation as a measured trade rather than free compression: how much of the teacher's advantage, and of a ten-model ensemble's gain, one small model actually kept.

Key takeaways