Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation

AI summary

Overview

• Research area: Natural Language Processing, specifically knowledge distillation (KD) for machine translation (MT) evaluated through environmental life cycle accounting.
• Technical level: Intermediate. The core argument is conceptual, but the paper assumes familiarity with teacher–student distillation, Transformer architectures, COMET scores, and carbon accounting units (CO2-equivalents).
• Scope: The paper measures whether distilling a smaller MT model from a larger teacher actually reduces carbon footprint, given that distillation adds upfront training and preprocessing costs that must be paid back by deployment volume.

What This Paper Is About

Knowledge distillation is usually sold as a way to make translation cheaper to run, because the student model is smaller than the teacher. The authors argue this framing is incomplete: distillation adds costs of its own (teacher training, teacher decoding or logit extraction, student training) that are fixed and paid upfront, while the savings only accumulate at inference time. The paper re-evaluates representative KD methods for machine translation by combining translation quality (COMET) with a full life cycle carbon accounting, and asks at what served volume, if any, distillation becomes the greener option.

Key Contributions

1. A life cycle aware evaluation protocol for KD in MT that decomposes environmental impact into teacher training, distillation, and inference, covering both operational emissions and amortized embodied hardware emissions using the Machine Learning Life Cycle Assessment (MLCA) framework of Morand et al. (2024).
2. A benchmark of representative KD methods evaluated jointly on translation quality and carbon impact, showing stark differences in carbon impact between methods that reach equal performance. Methods covered: Word-KD variants Sel-KD (Wang et al., 2021) and TIE-KD (Zhang et al., 2023), and Seq-KD variants Seq-Inter (Kim and Rush, 2016) and Seq-Rep (Zhang et al., 2018).
3. Evaluation across student capacity (a 65M student and a 16M student), language pair (EN→IS, EN→HI, EN→ES), corpus size (23M, 13M, and 6M sentence pairs), and teacher-cost accounting, using both bespoke encoder–decoder Transformer models and decoder-only LLMs.
4. Quantified deployment-volume break-even points against the teacher, showing that these points are serving-dependent and can shift by several orders of magnitude under batching.

Main Findings

• KD is not automatically greener: The paper's central claim is that distillation is only environmentally beneficial if the student serves enough volume to offset its production footprint. At low served volume, teacher training and distillation costs dominate the footprint; at larger volume, inference costs dominate and cheap student inference can offset distillation overhead.
• Method choice depends on setting, not universal dominance: No KD method is universally best. Word-level methods generally provide the best footprint–quality trade-offs for the 65M student, often reaching teacher-level quality, while sequence-level methods can become more competitive for the 16M student when synthetic targets simplify the learning problem. Dataset size does not just rescale the footprint, it can change which KD family is competitive.
• Aggressive compression hits a quality ceiling: The 16M student breaks even earlier than the 65M student because of its lower inference slope, but this comes with a stronger quality ceiling. On EN→IS, the 16M student gains little over the No-KD baseline and remains below the teacher; additional distillation yields diminishing returns.
• The No-KD baseline is usually the lowest-footprint option: It avoids teacher training allocation and distillation overhead, so KD is only justified when its quality gain over No-KD is meaningful. Marginal COMET improvements may be unattractive when the No-KD model already approaches the required quality level.
• Batching shifts break-even by orders of magnitude: Under sentence-level inference, break-even occurs at relatively modest volumes. Under batching, break-even moves into the hundreds of millions or billions of tokens, because batching improves both teacher and student inference and shrinks the student's relative per-token advantage. For EN→IS, reported break-even volumes include 386.97M tokens for Seq-Inter, 285.07M for Word-KD, and 733.37M for TIE-KD at batch size 256, rising to 1.60B, 1.18B, and 2.51B respectively at batch size 16000. The paper notes the trend is not perfectly monotonic across batch sizes because measured utilization and runtime vary.
• Teachers and students can have no break-even at all: Because KD and No-KD deploy the same student architecture, they have identical per-token inference footprints and therefore no deployment-volume break-even. Their comparison is quality-constrained instead: KD is justified only if its extra production footprint buys the required quality gain.
• The LLM case study supports the same logic under more favourable assumptions: With sunk teacher-cost accounting (λ_T = 0), distillation improved over the Qwen3-0.6B baseline in all settings. On EN→IS, where the baseline is weak, gains were large (for example, Qwen3-8B Seq-KD reached 48.18 and Word-KD 47.99 against a base of 28.30). Notably, the Qwen3-4B Word-KD student outperformed its teacher on EN→IS (46.36 vs 42.41), which the authors suggest may reflect task-specific adaptation and the regularizing effect of KD rather than greater general capability.
• Offline Word-KD is not uniformly preferable in the LLM setting: It provides richer token-level supervision than Seq-KD but requires offline teacher-distribution extraction and leads to a higher student-training footprint. For Qwen3-8B on EN→IS, offline Word-KD cost more while giving similar COMET to Seq-KD (47.99 vs 48.18).
• Access constraints shift environmental burden: The analysis assumes full access to the teacher. For closed-source or hosted models, logits are unavailable and teacher queries are rate-limited, making Seq-KD variants the only viable option even though they carry higher upfront emissions. The authors frame this as shifting part of the environmental burden from algorithmic choice to access policy.

Methodology in Plain English

The authors treat distillation as an accounting problem rather than only a quality problem. They define a functional unit that asks what the impact is of producing an MT system that serves a given volume of translation requests or decoded target tokens over one year at a specified quality level, and a system boundary restricted to the data-centre server side of the pipeline, excluding end-user devices, network transfer, and data acquisition or storage.

For each phase (training, distillation, inference) they estimate footprint as operational emissions, computed from electricity use, data-centre power usage effectiveness, and the grid emission factor, plus amortized embodied emissions, computed from the manufacturing footprint of the hardware scaled by phase runtime, hardware lifetime, and active utilization rate. Total footprint is the sum over phases and devices. A key modelling choice is teacher-cost allocation: the main MT experiments use λ_T = 1, assigning the full teacher training footprint to the distilled system, while the LLM case study uses λ_T = 0 because the pretraining footprint of LLM teachers is unavailable.

Because total footprint is written as a fixed production term plus a per-token inference slope multiplied by served volume, the break-even volume between a student and a teacher can be solved directly. The authors train a TF-Big teacher and two student sizes, TF-Base and TF-Tiny, on WMT 2024 General MT constrained track data for three translation directions with differently sized corpora, and compare a No-KD baseline, a teacher-only baseline, and KD pipelines. All runs use a single NVIDIA V100 GPU, with wall-clock time, utilization, and energy logged per phase. Quality is measured with COMET on FLORES+ devtest (using Unbabel/wmt22-comet-da). For inference, they first measure a simplified sentence-at-a-time serving setting, then measure footprint over fixed workloads of 32K–1M target tokens across multiple batch sizes, repeating each setting three times, and fit footprint–volume curves to compute break-even against the teacher. The LLM case study distils Qwen3-8B and Qwen3-4B teachers into a Qwen3-0.6B student on EN→IS and EN→ES using 100K sampled sentence pairs, with offline Word-KD in which teacher token distributions are precomputed and reused.

Why This Matters

Impact on research. The paper argues that evaluations reporting only translation quality, or only inference efficiency, can lead to systematically different conclusions under life cycle-aware evaluation. It challenges the implicit assumption that a smaller student equals a greener system, and proposes that KD be benchmarked under an explicit functional unit tied to served volume, with production footprint, per-token inference slope, and break-even volume reported. It also points out that prior work (for example Zhang et al., 2023; Niemann et al., 2025; Rafat et al., 2023; Yuan et al., 2024) quantifies only subsets of the distillation and deployment life cycle.

Real-world applications:

• One-off translation workloads such as translating a single document or a single novel-scale amount of text, where the paper concludes KD is rarely justified from an emissions perspective.
• Sustained, high-throughput translation services or repeated deployments, where KD can reduce life cycle emissions while meeting a quality constraint.
• Model selection under sustainability constraints, where practitioners choose checkpoints from a footprint–quality Pareto frontier rather than from quality alone.
• Deployment planning on restricted or hosted infrastructure, where the serving configuration (batched or unbatched) determines whether the break-even volume is reachable.

Industry relevance. Break-even volume is described as serving-dependent rather than an intrinsic property of a distilled model, so the same student can be a net emissions win or loss depending on batching and utilization in the production stack. The paper also raises a policy dimension: when teacher logits are unavailable from closed-source providers, practitioners may be forced toward higher-emission sequence-level distillation, shifting part of the environmental burden from algorithmic choice to access policy.

Future Directions

1. Repeating the full experimental matrix across multiple training seeds, since each KD configuration was trained once and the reported paired-bootstrap confidence intervals over COMET do not capture variation from initialization or data ordering.
2. Investigating synthetic-data filtering for Seq-KD, with the caveat that filtering is only useful from a life cycle perspective if it reduces the amount of teacher generation required, since filtering after the full synthetic corpus has been generated does not remove the main decoding footprint.
3. Quantifying and attributing the constrained-choice penalty that arises when closed-source or rate-limited teachers make word-level KD infeasible.
4. Extending the accounting framework beyond the settings studied here, since the framework is not MT-specific but production and inference costs must be re-estimated for each task and deployment regime, and addressing the study's limited architecture coverage and its exclusion of truly low-resource MT settings.

Target Audience

Researchers and practitioners in machine translation and model compression who select or evaluate distillation methods; sustainability and Green AI researchers interested in life cycle assessment for ML systems; ML engineers making deployment decisions about whether to distil, which distillation family to use, and what serving volume is required to justify it; and infrastructure or policy stakeholders concerned with how model access constraints affect the environmental cost of deployed systems.

Authors’ abstract

Knowledge distillation (KD) is a technique to compress a larger teacher system into a smaller student. In machine translation, KD is commonly evaluated through translation quality and inference efficiency, without jointly accounting for the environmental costs of producing and deploying the distilled system. We evaluate representative KD methods both on bespoke MT models and LLMs, by considering both translation quality and computational cost, using the Machine Learning Life Cycle Assessment tool, which accounts for costs throughout the KD model life cycle. Our key finding is that the deployment volume required to amortize KD is serving-dependent and can shift by several orders of magnitude under batching. We include actionable guidance for selecting, developing, and evaluating KD methods under quality and compute-induced constraints.