Long-Form ASR, Chunking, and Non-Speech Hallucinations

A stretch of silence can come back as a confident paragraph. That is measured, not anecdotal. A 2024 FAccT study, Careless Whisper, ran audio through the Whisper API and then measured how much of each file was non-vocal, using pyannote voice activity detection. The relationship runs the way the mechanism predicts: “Audio files yielding hallucinations tended to have higher non-vocal shares of durations (29% versus 26% of total durations).”

The disparity underneath is the part worth sitting with. Aphasia speakers in that data had a 41% non-vocal share. Controls had 15%. The gap is reported at p < 2.2e-16. The hallucination rate tracked it too: 1.7% of aphasia segments against 1.2% of control segments. The people whose speech carries the longest pauses are the people whose transcripts acquire the most text that was never said.

What comes out reads like ordinary speech. So does the text repeated where one chunk overlaps the next. A review that stops at aggregate error and insertion counts will catch neither. That is the difficulty this whole lesson circles. The characteristic failures of long-form transcription are fluent, and fluent text is exactly what error counts are worst at flagging.

Thirty seconds is the unit the model was built for. An hour of audio is the unit it is asked to serve. OpenAI trained Whisper on 680,000 hours of audio, cut into 30-second segments. Long-form ASR fills the gap between that unit and the recording. It breaks continuous audio into windows small enough to decode, and tries to preserve context, timestamps, speaker changes and boundaries across the seams. Chunk length, overlap, prompt carryover, reset policy and non-speech handling are all settings someone chose. Each of them can produce omissions, duplicates, drift or unsupported text.

Where the next window starts is itself model output. The 2023 WhisperX paper names the consequence in one line: “Such a method is prone to severe drifting since timestamp inaccuracies in one window can accumulate to subsequent windows.”

A larger model does not fix this. The mechanism is not a shortage of capacity. It is where the audio was cut. WhisperX's Table 3 prices that directly: feeding full audio in batches of 32, with no VAD pre-segmentation, raises TED-LIUM word error rate from 10.52 to 78.78. Same model, same recordings, same decoder. The only thing changed is the boundaries, and the transcript goes from usable to worthless. Boundaries are detected first, and the long-form controls then run over the chunks those boundaries define. Miss a speaker change, or end a segment mid-sentence, and the controls are being applied to a window that was already wrong. Tuning them does not recover 78.78.

Which setting breaks is not the same in every recording. Meetings, broadcast archives, podcasts and body-worn audio defeat segmentation in four different ways. Each is counted in its own unit: a duplicated seam in words, a drifting timestamp in seconds, an invented passage in the harm it carries. The non-vocal-share evidence that settles the silence case says nothing about the other three. A WER of 10.52 or 78.78 says nothing about which phrases were invented.

Which of those four you are serving decides how much context a chunk should carry, and there are three answers. Independent chunks limit error propagation and give up cross-boundary language context to do it. Context-carrying chunks keep that context and accept the propagation. Hierarchical processing takes the same trade at a different price again.

The middle branch is not hypothetical, and neither is the resolution. WhisperX's default configuration sets `Condition on previous text` to False, and its authors say why: “In practice, we find this restriction to be beneficial, since conditioning on previous text is more prone to hallucination and repetition.” A published system, on a long-form benchmark, threw away cross-boundary context rather than carry the hallucination and repetition that came with it. That same prior-text conditioning is the mechanism behind the invented text the FAccT study counted.

That is one team's resolution on one workload, not a general verdict. The three branches answer related but different questions, and no single measurement settles all three.

Choosing between them is only one of four choices sitting behind a long-form transcript: measurement, modeling, decision and verification. A single number computed over a whole recording hides all four. The path runs in the order the choices are actually made. What gets measured off the audio, how the windows are modeled, which reset policy is decided on, how the result is verified. The order is the point, because the earliest choice is the one nobody goes back to.

Step three, managing context state, is where the reset policy lives. OpenAI's own numbers show why it cannot be settled with an average. The Whisper paper describes the coupling: “Transcribing long-form audio using Whisper relies on accurate prediction of the timestamp tokens to determine the amount to shift the model's 30-second audio context window by, and inaccurate transcription in one window may negatively impact transcription in the subsequent windows.”

Its Table 5 then prices the knob. Conditioning on previous text cuts average long-form WER across seven datasets from 10.2 to 10.0. On the Meanwhile set it raises WER from 4.61 to 6.16. The same setting helps on the mean and hurts a slice. Two tenths gained in the reported average, bought with a point and a half lost on one recording type. Whichever way you set it, report the slice, not only the mean.

Verification is the step worth building first. Its cheapest form is a torture set: audio assembled to be handed to someone hostile, who should then be able to demonstrate repeated text at a chunk overlap within a minute of looking. Fill it from the four recording types — silence, music, crosstalk, speaker changes, applause, domain shifts. Make some of the files long enough that the drift has room to compound.

"Long enough" is a published number, not a feeling. Earnings-21, released in 2021, is 44 files of 2020 earnings calls from nine sectors, 39 hours and 15 minutes in total. Of the individual recordings its authors write: “The recordings in this corpus range in length from less than 17 minutes to 1 hour and 34 minutes with the average recording being about 54 minutes in length.” Earnings-21 is also one of the seven long-form sets behind Whisper's own Table 5. Build a torture set at that scale and you are testing at the duration the published averages were computed over. Fifty-four minutes of audio is a long chain of thirty-second windows, each one shifted by the last one's timestamps.

Three notes go with the set, and they are the notes the next team will need. Write down what your boundary detection assumes about the audio. Write down one recording where that assumption is wrong. Write down what the long-form controls then do to that recording. That is the part nobody records.

The torture set shows what breaks. A report has to say how often, and one number cannot. Word error rate and insertion rate by recording duration is the right place to start. It will not by itself show repeated text at the chunk overlaps, because a duplicated seam is fluent, correctly spelled and locally plausible.

Two systems can post the same WER and invent different amounts of text. That has been demonstrated, not merely suspected. Frieske and Shi define ASR hallucinations as fluent but semantically unrelated output, and in January 2024 they reported what that does to the metric everyone quotes: “We show that commonly used metrics, such as word error rates, cannot differentiate between hallucinatory and non-hallucinatory models.”

So say what unit each figure is in, which recordings it covers, how uncertain it is and how long they ran. Then put beside it the measurements that do see seams, drift and invented speech.