Listen Like a Teacher: Mitigating Whisper Hallucinations using Adaptive Layer Attention and Knowledge Distillation

AI summary

Overview

Research area: Automatic speech recognition (ASR), specifically hallucination mitigation and noise robustness in OpenAI's Whisper speech model.

Technical level: Intermediate to Advanced. The paper assumes familiarity with transformer encoder-decoder architectures, multi-head attention, knowledge distillation, and metrics such as WER and SNR.

Scope: The paper proposes and evaluates a two-stage modification to Whisper-small — Adaptive Layer Attention (ALA) on the encoder plus Multi-Objective Knowledge Distillation (MOKD) from a clean-speech teacher — and benchmarks it against three baselines in Arabic, French, Hindi, and English across signal-to-noise ratios from -10 dB to clean speech.

What This Paper Is About

Whisper is a widely used open-source ASR system that performs well across languages and zero-shot settings, but it frequently produces hallucinations — fluent yet semantically incorrect transcriptions — particularly in noisy audio or non-speech segments. Most prior fixes work outside the model: audio preprocessing, post-processing transcriptions, or data augmentation. This paper instead modifies Whisper's internals, first by fusing information from multiple encoder layers so the model does not depend on a single noise-distorted layer, and second by training the model on noisy audio while forcing it to imitate a clean-audio teacher's internal representations and attention patterns.

Key Contributions

1. Adaptive Layer Attention (ALA) over the Whisper encoder. Encoder layers are grouped into semantically coherent blocks by inter-layer correlation analysis, and a learnable multi-head attention module dynamically fuses the block representations so the model jointly exploits low- and high-level features.

2. A multi-objective knowledge distillation (MOKD) strategy. The student is optimised with cross-entropy loss, encoder cosine similarity loss, decoder cosine similarity loss, and mean squared error on the decoder's cross-attention maps, aligning it with a teacher trained on clean speech.

3. A deliberately asymmetric teacher-student setup. The student trains exclusively on noisy data (SNR -8 dB to +4 dB) while distilling from a clean-data teacher; the authors report that using a noisy teacher resulted in inferior performance.

4. A combined two-stage framework evaluated across four languages and seven noise/clean conditions. The paper reports WER and SeMaScore for every configuration, plus ablations of encoder-fusion strategies and individual KD loss terms.

Main Findings

• ALA improves over both Whisper baselines at every noise level and language. In Table 1, average WER/SeMaScore for the W-ALA model is 59.44/0.8415 (Arabic), 30.75/0.8930 (French), 19.64/0.9284 (Hindi), and 9.68/0.9581 (English). The corresponding Baseline-2 (Whisper-small fine-tuned on the noisy training set) averages are 63.42/0.8083, 33.08/0.8837, 21.44/0.9178, and 12.46/0.9567. Baseline-1 (off-the-shelf Whisper-small) is far worse, averaging 230.53/0.0815, 127.94/0.3696, 141.08/0.0911, and 12.97/0.9434.

• The gains are largest at severe noise. At -10 dB SNR for English, W-ALA reaches 29.68 WER / 0.8772 SeMaScore versus 39.64/0.8763 for Baseline-2. At -5 dB, French improves from 39.29/0.8521 (Baseline-2) to 37.15/0.8645 (W-ALA).

• MOKD outperforms W-ALA, Baseline-2, and the Distil-Whisper baseline. Average WER/SeMaScore for W-MOKD is 57.40/0.8557 (Arabic), 28.18/0.8965 (French), 18.61/0.9432 (Hindi), and 8.56/0.9690 (English). At -10 dB English, W-MOKD reaches 26.43/0.9046; at clean English it reaches 3.18/0.9836.

• The Distil-Whisper baseline (Baseline-3) performs poorly on this setup. Its average WER/SeMaScore is 69.74/0.6639 (Arabic), 40.60/0.8181 (French), 51.39/0.7544 (Hindi), and 50.36/0.8592 (English) — worse than Baseline-2 in every language despite using the same teacher, because only two decoder layers are retained and the encoder is frozen.

• Encoder attention concentrates on the earliest block under noise. In Figure 3(b), Block 0 receives an average attention weight of 0.586, while Blocks 1 and 2 receive "significantly lower and nearly equal attention." The frame-wise heatmap shows Block 0 dominating especially in noise-affected segments.

• Mean pooling is the best block-fusion strategy. In the Stage-1 ablation on Hindi (Table 4), using multi-head attention over mean representations of each block gives 40.74/0.8257 at -10 dB and 11.41/0.9669 on clean speech, beating weighted sum (75.85/0.4822; 29.56/0.7521), frozen MHA over all layers (52.73/0.5454; 15.62/0.8928), trainable MHA over all layers (45.12/0.6902; 14.87/0.9290), and the three sampled-layer configurations (1,7,12: 42.23/0.7032; 14.20/0.9352 — 6,11,12: 41.91/0.7105; 12.16/0.9480 — 3,9,12: 41.53/0.7135; 12.14/0.9492).

• The best distillation combination is final-layer cosine similarity plus decoder cross-attention MSE plus ALA. In the Stage-2 ablation on Hindi (Table 5), CE alone gives 42.77/0.8027 at -10 dB and 12.77/0.9638 on clean; final-layer cosine similarity alone gives 42.97/0.8246 and 12.37/0.9622; adding ALA gives the best rows in the table at 38.13/0.8455 and 11.23/0.9675. Distance-based losses hurt badly — IRLE on final layers drops to 61.14/0.6102 and 39.36/0.7333.

• The overhead is small. Against Baseline-2 on LibriSpeech English test-clean at batch size 1, W-ALA increases latency by 8% (140 ± 10 ms to 152 ± 11 ms), real-time factor by 9% (0.021 to 0.023), and peak VRAM by approximately 1 GB (1.5 GB to 2.6 GB), while adding 0.98% more parameters.

• Data-consistency note: the Arabic Baseline-2 row in Table 3 is numerically identical to the English Baseline-2 row in the same table (average 12.46/0.9567), and does not match the Arabic Baseline-2 average of 63.42/0.8083 reported in Table 1. The French and Hindi Baseline-2 rows in Table 3 do match Table 1. Readers should treat the Arabic Baseline-2 comparison in Table 3 with caution.

• Hallucination is not reported as a separate rate. The paper states that hallucinations are measured through WER and SeMaScore, and describes reductions qualitatively; no standalone hallucination rate (for example, hallucination error rate) is reported for any model.

Methodology in Plain English

Whisper's encoder is a stack of layers; each layer captures a different level of detail, from raw acoustics to meaning. Normally only the last layer's output is handed to the decoder. The authors first measure how similar the layers' outputs are to each other using cosine similarity. On a subset of the Hindi Kathbath test set they find three natural groupings: layers L1–L6 (low-level acoustic features), L7–L11 (higher-level semantic abstractions), and L12, which diverges from everything else and is described as possibly overfitting to noise. The paper states this structural pattern held across all evaluated languages.

ALA then averages the layers inside each group to make one summary representation per group, adds positional encoding so time order is preserved, and lets a trainable multi-head attention module (6 heads) decide, for every token, how much to draw from each group — using the final encoder layer's state as the query. The attention output is added back to that query and normalised before going to the decoder.

Stage two adds a teacher. The teacher is a Whisper-small model trained on clean speech; the student is the ALA-augmented model trained on noisy speech (SNR -8 dB to +4 dB, with noise drawn from the DEMAND database and 1% noise-only samples labelled with empty transcripts). Four losses are summed with weights: cross-entropy against the ground-truth transcript, cosine similarity between teacher and student encoder states at the final layer, cosine similarity between their decoder states at the final layer, and mean squared error between their decoder cross-attention maps. A grid search over weights from 0.5 to 2.0 found λ1 = 0.8 with the others at 1.0 worked best.

Training used NVIDIA H100 GPUs: 1 GPU for ALA (15 epochs, learning rate 5×10⁻⁵ for base parameters and 9×10⁻⁵ for ALA parameters, 5000 warm-up steps, cosine decay) and 4 GPUs in distributed fashion for MOKD (72,000 steps, learning rate 1×10⁻⁵, 100 warm-up steps, linear decay). Whisper-small was chosen as the base model because prior work reports smaller Whisper variants have higher WER and hallucination rates. Data came from Kathbath (Hindi), CommonVoice-15 (Arabic, French), and LibriSpeech-100 with the test-clean split (English); audio longer than 30 seconds was chunked to at most 30 seconds.

Why This Matters

The paper argues that hallucinations originate in misaligned internal representations rather than being purely an output-filtering problem, and it shows that modifying the model itself — rather than only preprocessing audio or post-processing text — can improve both WER and a semantic metric across four languages and a wide SNR range. If the reported gains hold, this is a template for making any transformer-based speech model more reliable in noise without retraining on clean data. The measured cost is modest: 0.98% more parameters, 8% more latency, and roughly 1 GB more peak VRAM, which keeps it viable on modern accelerators.

Real-world applications:

• Voice interfaces in noisy environments — in-car assistants, kiosks, and smart speakers where microphone input is degraded by road, crowd, or machinery noise.
• Automatic captioning and transcription services — meeting and broadcast transcription, where fluent-but-wrong hallucinated text is more damaging than an obvious error because it escapes standard quality checks.
• Call centre and customer-service analytics — large-scale analysis of recorded calls with highly variable acoustic quality.
• Accessibility tooling — live captioning for deaf and hard-of-hearing users, where semantic fidelity under noise matters more than raw token accuracy.

Industry relevance: The work comes from Sony Research India and targets deployed ASR, which explains the emphasis on latency, real-time factor, and peak memory on H100 GPUs alongside accuracy. Distilling a clean-speech teacher into a noisy-input student is also relevant to any organisation that has clean-reference recordings but deploys in noisy conditions.

Future Directions

1. Cross-lingual generalisation — the authors explicitly list this as future work, though the paper already evaluates four languages and reports the encoder similarity pattern is consistent across them.

2. Applying the approach to other transformer-based speech models — also named by the authors; ALA and MOKD are described in a Whisper-specific way but are not inherently Whisper-only.

3. Stronger or better-matched distillation baselines — the Distil-Whisper baseline (Baseline-3) performed poorly here with a frozen encoder and two decoder layers, so it is unclear how the method compares against other distillation schemes that align intermediate layers, such as the attention-alignment methods cited in the related work.

4. Reporting hallucinations directly — the paper relies on WER and SeMaScore rather than a hallucination rate. A dedicated hallucination metric, and evaluation on noise-only or non-speech segments, would

Authors’ abstract

The Whisper model, an open-source automatic speech recognition system, is widely adopted for its strong performance across multilingual and zero-shot settings. However, it frequently suffers from hallucination errors, especially under noisy acoustic conditions. Previous works to reduce hallucinations in Whisper-style ASR systems have primarily focused on audio preprocessing or post-processing of transcriptions to filter out erroneous content. However, modifications to the Whisper model itself remain largely unexplored to mitigate hallucinations directly. To address this challenge, we present a two-stage architecture that first enhances encoder robustness through Adaptive Layer Attention (ALA) and further suppresses hallucinations using a multi-objective knowledge distillation (KD) framework. In the first stage, ALA groups encoder layers into semantically coherent blocks via inter-layer correlation analysis. A learnable multi-head attention module then fuses these block representations, enabling the model to jointly exploit low- and high-level features for more robust encoding. In the second stage, our KD framework trains the student model on noisy audio to align its semantic and attention distributions with a teacher model processing clean inputs. Our experiments on noisy speech benchmarks show notable reductions in hallucinations and word error rates, while preserving performance on clean speech. Together, ALA and KD offer a principled strategy to improve Whisper's reliability under real-world noisy conditions.