Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery

AI summary

Overview

Research area: Efficient machine learning / model compression for automatic speech recognition (ASR), specifically encoder layer pruning in OpenAI's Whisper model.

Technical level: Intermediate. The paper assumes familiarity with transformer encoder-decoder architectures, knowledge distillation, and Word Error Rate as an ASR metric, but its core idea — delete the least useful layers — is conceptually simple.

Scope: The paper develops a WER-guided method for removing six entire encoder layers from whisper-large-v3-turbo, recovers lost accuracy with label-free knowledge distillation, and benchmarks the resulting model's size and inference speed on consumer hardware.

What This Paper Is About

Prior compression work on Whisper has focused almost entirely on the decoder, producing variants like whisper-large-v3-turbo (32 decoder layers reduced to 4) and Distill-Whisper (reduced to 2). The encoder has received far less attention, and existing encoder approaches have not seen wide adoption, which the authors suggest may be due to the need for custom inference implementations. This paper asks whether whole encoder layers can simply be deleted — yielding a shallower network that runs on existing inference libraries without code changes — and whether the resulting accuracy loss can be recovered by training on unlabeled audio.

Key Contributions

1. A simple layer-pruning method that shrinks whisper-large-v3-turbo's encoder stack by 18.5%, deployable through existing inference libraries without architectural changes or custom code.
2. A demonstration that multilingual performance lost through zero-shot layer pruning can largely be recovered through label-free knowledge distillation at a modest training and data budget (2000 steps, roughly half an hour on an Nvidia A100 GPU).
3. Evidence that data-driven layer selection is essential, established by comparing the selected set against 50 random selections of six layers.
4. Release of the code and the pruned model as public artifacts.

Main Findings

• Accuracy after pruning and recovery: Zero-shot pruning raises mean WER across four languages from a baseline of 18.2% to 21.9% (an increase of 3.8 percentage points). Label-free distillation recovers this to 20.1% (an increase of 1.9 percentage points over baseline).

• Per-language results: Danish moves from 23.9 to 32.1 (+8.2) zero-shot and recovers to 27.3 (+3.4). English moves from 15.4 to 16.6 (+1.2) and recovers to 16.1 (+0.7). German moves from 17.1 to 18.3 (+1.2) and recovers to 18.1 (+1.0). French moves from 16.3 to 20.7 (+4.4) and recovers to 18.7 (+2.5).

• Which layers were removed: The six pruned layers are layers 5, 6, 7, 9, 11 and 12, all in the start-to-middle of the encoder. Layers 0 and 31 are described as critical, causing catastrophic degradation upon removal.

• Layer selection matters greatly: Against 50 random selections of six layers drawn from layers 1–15, random selection spans from +5 to +471 WER, compared to +3.8 for the ΔWER-guided selected set.

• A sharp pruning boundary: Degradation is modest and roughly linear up to k=6, after which performance collapses sharply at k=7. Even the most benign seventh layer (L8) causes 2.6 times more degradation than the optimal k=6 selection. The optimal seventh layer, L8, would create a contiguous five-layer gap from L5 to L9.

• Model size reduction: Removing six layers eliminates 118M encoder parameters, reducing the encoder from 637M to 519M parameters and the full model from 1543 MB to 1318 MB in bfloat16 precision — a saving of 225 MB with no change to the decoder.

• Inference speedups: On an Apple M4 Pro with the Transformers library and Apple's MPS backend, encoder speedups are 1.22x at batch size 1 and 1.24x at batch size 8. End-to-end speedups are 1.08x at batch size 1 and 1.75x at batch size 8. At batch size 8, the encoder accounts for 56% of full-model time versus 36% at batch size 1.

• Cross-lingual recovery from monolingual data: Distillation used English-only audio, yet all four languages recovered, including Danish and French, which degraded the most. The authors interpret this as evidence that MSE distillation on hidden states recovers acoustic representations rather than language-specific features.

Methodology in Plain English

The researchers start from the observation that Whisper's encoder uses residual connections, so the signal carried through the stack stays highly similar from layer to layer — suggesting some layers may be redundant. Rather than guessing, they measure each layer's importance empirically: they run the full model on the FLEURS test set, then run 32 separate versions of the model, each with exactly one different encoder layer deleted, and record how much the Word Error Rate changes. Averaging this "leave-one-layer-out" change across Danish, English, German, and French gives each layer an importance score.

The six layers with the smallest impact — layers 5, 6, 7, 9, 11 and 12 — are then removed permanently. Removal is implemented simply by replacing the encoder's PyTorch nn.ModuleList with a copy that omits those layers, so the result is an ordinary shallower model that any standard inference library can run.

To recover the accuracy lost by this zero-shot deletion, the authors treat the original encoder as a teacher and the pruned encoder as a student. They freeze the full encoder and the decoder, and train only the pruned encoder to produce hidden states that match the teacher's, minimizing a Mean Squared Error loss between the two sets of hidden states. Training uses the AdamW optimizer with a batch size of 8 for 2000 steps on the validation split of the English-only People's Speech dataset — no transcript labels are needed, only audio. The paper also quantifies the benefit of the ranking procedure by comparing against 50 random six-layer selections, and probes the limits of pruning by extending the same ranking approach to k=7 through k=14.

Why This Matters

Impact on research: The paper extends the compression literature beyond the decoder to the encoder, which the authors note has been relatively overlooked. It provides a concrete data point that encoder layers in Whisper are partly redundant, and it questions whether expensive multilingual WER-based ranking is worth the cost — concluding that it is, given the wide spread of random-selection outcomes.

Real-world applications:

• Medical transcription, one of the application areas the authors cite, where on-device or on-premise ASR reduces reliance on cloud inference.
• Real-time captioning, where the 1.75x end-to-end speedup at larger batch sizes directly increases throughput.
• Deployment on resource-constrained devices, where the 225 MB bfloat16 size reduction matters for storage and memory budgets.
• Multilingual transcription services, since the pruned model retains performance across Danish, English, German and French.

Industry relevance: The pruned model is a drop-in replacement requiring no changes to inference frameworks, which removes a major practical barrier that the authors suggest has limited adoption of prior encoder-compression approaches. Speedups were measured on consumer hardware (an Apple M4 Pro) using the widely used Transformers library, indicating relevance to everyday deployment rather than only to data-center settings.

Future Directions

• Explain the cliff phenomenon. The paper calls for further research into why performance degrades only modestly up to k=6 but collapses sharply beyond it, and why the optimal seventh layer (L8) is adjacent to the existing pruned block.
• Broaden the language sweep. The authors limited themselves to four languages and note that a more thorough sweep might reveal pitfalls in low-resource languages not covered here.
• Test multilingual distillation data. Distillation used English-only audio; the authors ask whether multilingual data would further improve recovery for the languages hit hardest — particularly Danish and French.
• Generalize to newer ASR models. The findings are limited to whisper-large-v3-turbo's encoder stack, and the authors suggest investigating whether they transfer to more modern transcription models such as Cohere Transcribe.

Target Audience

This paper is most useful to machine learning engineers and researchers working on model compression and efficient inference, particularly those deploying ASR systems on constrained or on-premise hardware. It also suits practitioners who want a practical, drop-in way to shrink Whisper without writing custom inference code, and researchers studying transformer redundancy and layer importance in encoder-decoder architectures. Readers without background in ASR or transformer internals will find the core idea accessible, though the evaluation details assume familiarity with WER and knowledge distillation.

Authors’ abstract

Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to $18.5\%$ of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to $20.1\%$ after distillation, compared to $21.9\%$ zero-shot, going from a baseline of $18.2\%$. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).