Attention Encoder–Decoder Speech Recognition

In 2023 a team pushed 13,140 AphasiaBank audio segments through the Whisper API. They went looking for text that was not in the audio at all. They found 187 segments that reliably hallucinated. On average 1.4% of transcriptions contained hallucinations. Their 2024 paper opens with the finding: "While many of Whisper's transcriptions were highly accurate, we find that roughly 1% of audio transcriptions contained entire hallucinated phrases or sentences which did not exist in any form in the underlying audio."

The rest of the count is what makes this a lesson rather than a curiosity. Of the 312 hallucinated transcriptions, 38% contained at least one explicit harm: 19% perpetuating violence, 13% inaccurate associations, 8% false authority. The burden was not spread evenly. The hallucination rate was 1.7% for speakers with aphasia against 1.2% for the control group. The invented text arrived most often for the speakers whose audio was hardest. It arrived for the speakers whose transcripts were least likely to be contested.

That is the failure this lesson is built around. Its name is worth fixing early: fluent continuation unsupported by audio. It is hard to catch because the usual report can be complete and still miss it. WER can be published in full, with deletion, insertion and repetition profiles broken out. An invented clause still sits inside the sentence, looking exactly like the words that were spoken.

The mechanism behind those hallucinated sentences is the architecture itself. An attention encoder–decoder maps audio to contextual encoder states. It then generates output tokens autoregressively while attending to those states. The difference from CTC is that the decoder conditions explicitly on the output history it has already produced. It knows what it just wrote, and it writes the next token partly to fit what came before. That is what buys flexible alignment, and flexible alignment is a real advantage on ordinary speech.

It is also the entire cost. And the cost is not a critic's characterisation. It sits on the limitations list of the system that was measured above. Whisper's own paper, under improved decoding strategies, writes: "They are a combination of failure modes of seq2seq models, language models, and text-audio alignment and include problems such as getting stuck in repeat loops, not transcribing the first or last few words of an audio segment, or complete hallucination where the model will output a transcript entirely unrelated to the actual audio." Skip, repeat, invent. Those three failures were named in 2022. They were named by the people who trained the model on 680,000 hours of labeled audio, about their own deployed system.

All three arrive most readily on long, noisy or out-of-domain audio. Standard full-context attention carries a second limit as well: it is not inherently streaming.

Work on such a system runs from encoding the utterance at one end to searching and constraining the output at the other. Name an owner at either end. Otherwise the search inherits assumptions that nobody is answerable for.

The path from audio to transcript has three stages and one dependency worth stating plainly. Encoding the utterance produces the contextual states. Attending from the decoder consumes them. Then comes searching and constraining — beam width, coverage terms, any fusion with a language model. That is where a team finds out whether the assumptions behind the first stage held.

That last stage is not a tuning detail, and there is a published ledger for it. Whisper introduces the first constraint by naming the pathology it exists to suppress: "First, we use beam search with 5 beams using the log probability as the score function, to reduce repetition looping which happens more frequently in greedy decoding." A table in the same paper then prices the whole stack across seven long-form English datasets. Average WER was 11.0 under greedy decoding, 10.6 with beam search added, 10.2 with voice activity detection, 10.0 with previous-text conditioning, and 10.0 again with the initial timestamp constraint. Temperature fallback belongs to the same set of additions.

One point of word error rate, bought by five stacked heuristics. One of them exists solely because the decoder loops. That is the shape of the trade. The constraints are load-bearing, they are cumulative, and the last one moved the average not at all. Which side of that trade is correct depends on what a wrong transcript costs once it leaves the system. Nothing in the search code knows that.

The decoder's freedom is not a defect waiting to be engineered away. It pays in some settings and costs in others. The four below differ mainly in how much audio the model has to hold at once before it commits to anything.

The hybrid CTC/attention paper put the diagnosis in one sentence in 2017: "Attention-based ASR, however, may be prone to include deletion and insertion errors (see Figure 3 and related discussions) because of its flexible alignment property, which can attend to any portion of the encoder state sequence to predict the next label." The advantage and the error mode are the same property.

Wherever a decoder writes its own history, the same four terms recur. They are one story rather than four definitions. Teacher forcing is how the decoder is trained. Exposure bias is what teacher forcing leaves behind. Coverage and beam search are the machinery built afterwards around that same gap. Coverage in particular has a documented origin. Chorowski and Jaitly built it as a fix for a named pathology, not as a general precaution.

Three of those pieces answer different questions, and they are routinely waved through as though one clearance covered all of them. Teacher forcing is a training-time choice: it conditions training on the reference history. Autoregressive inference is a run-time fact: the model conditions on the history it generated itself. Joint CTC–attention is an architectural answer to the distance between the two.

The gap itself was stated plainly in 2015. The scheduled sampling paper put it in its abstract: "At inference, the unknown previous token is then replaced by a token generated by the model itself. This discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence." The authors never use the term "exposure bias". They describe the mechanism and propose a curriculum against it. Scheduled sampling gradually replaces the reference previous token with the model's own generated token during training. The decoder then learns on the histories it will actually have.

Read the two failure descriptions together. Errors that accumulate quickly along the generated sequence, from 2015. Getting stuck in repeat loops and complete hallucination, from 2022. Evidence that a decoder trains stably says nothing about how it behaves on its own predictions. A joint model that scores well on clean audio has not been tested where the gap actually opens. None of the three is cleared by what clears another.

No single number carries that. WER is one of four things a release should report, even split into deletion, insertion and repetition profiles. It is also the one most likely to be quoted alone. Recall what happened to Listen, Attend and Spell. 14.1%, 10.3% and 8.0% were all word error rates on the same task. Knowing them tells you nothing about which of those transcripts invented a clause. The other three lines are what keep the first from carrying a release claim by itself.

Take one fluent continuation the audio never supported and trace it end to end. Write down what encoding the utterance assumed. Write down the one counterexample that breaks the assumption. Then write down what searching and constraining did with it once it was already wrong.

The published work gives you the shape of the exercise. The AphasiaBank study isolated 187 segments out of 13,140 that hallucinated reliably. Reliably means the failure reproduced, which is what makes a trace possible at all. Chorowski and Jaitly traced incomplete transcriptions back to the interaction between the EOS token's probability and the language model, and only then added a coverage term. Both start from one reproducible bad output rather than from an aggregate.

The value is not the individual fix. It is that another team can afterwards challenge a clean-reading transcript without being told that it reads fine.