ASR Decoding, Language Models, and Contextual Biasing

A recognizer's neural model does not decide the transcript on its own. Decoding is a search. It looks for the sequence that scores best under acoustic or sequence-model scores. Usually external language evidence, a lexicon, a length term and a contextual bias are in that sum too. Change the search settings or the priors and the errors change, with the neural model untouched. That is why this lesson exists. The thing tuned last week may be the thing that decided what the microphone is reported to have heard.

Two of those ingredients matter most, because everything later turns on them. An external language model can make a transcript more plausible while making it less faithful to the acoustics. That is a documented shortcoming, and the failure-modes section below quotes the paper that named it. Context lists and hotwords are the second. They are hypotheses about what is likely to be said, not ground truth about what was said. Scope them to a user, a task, a stretch of time, a document state, or they are live everywhere. So the release question comes in two halves. Oracle versus selected hypothesis error asks what the search could have found. Rare-entity precision and recall asks what it did with the words that actually matter. Neither one on its own is a release decision.

None of this used to be so hard to look at. Before neural decoders, the language model was a separate object you could inspect and swap. Kaldi composes a decoding graph called HCLG out of four transducers: H holds the HMM definitions, C the context dependency, L the lexicon and G the grammar or language model. The composition runs on OpenFst, a weighted finite-state transducer library. Biasing a recognizer toward a contact list meant editing G. That is a far more auditable operation than a prompt. The thing that changed was a named object with a shape. Someone else could be handed it and asked what it now allowed.

Work on a decoder has a shape. It starts by defining the search space: units, lexicon constraints, language model, allowed outputs. It ends by auditing what the search actually chose — oracle hypotheses against selected ones, beam errors, insertions, rare entities, runtime cost. Between those two sits the step that hides the most. That is deciding how the scores are combined: acoustic against language model, hypothesis against bias list, tuned with length and coverage terms on validation data. It is also the step teams skip.

The third step, activating context, is not a hack bolted onto a decoder. It is a designed mechanism with a published payoff, so the warning that follows lands against a real gain rather than a straw one. A system called CLAS, published in 2018, optimises the ASR model jointly with embeddings of context n-grams instead of bolting a bias list onto a finished system. Its five authors report: “Across a number of tasks, we find that the proposed CLAS system outperforms the baseline method by as much as 68% relative WER, indicating the advantage of joint optimization over individually trained components.” The baseline it beat was shallow-fusion contextual biasing. Sixty-eight per cent relative is why nobody is going to give context injection up.

Which is exactly why the scoping step matters. Skip score combination and the assumptions built into the search space travel intact all the way to the audit, where they get inspected rather than questioned. A boost that never expires is one of those assumptions. It was set once, and nobody revisited it. By the time anyone reads the transcripts it looks like part of how the system hears, not a choice somebody made on a Tuesday. Task-specific names, contacts, products and phrases need an expiry and a provenance the moment they are switched on. What follows is what their absence looks like when it reaches a hospital.

The rate has been measured. Roughly 1% of Whisper audio transcriptions contain entire hallucinated phrases or sentences that were never in the audio at all. Koenecke and four colleagues found that in 2024. Their abstract: “While many of Whisper's transcriptions were highly accurate, we find that roughly 1% of audio transcriptions contained entire hallucinated phrases or sentences which did not exist in any form in the underlying audio.” Not a substituted word. Not a garbled name. Whole sentences with no acoustic evidence behind them. And 38% of those hallucinations included explicit harms: perpetuating violence, inventing associations, or implying false authority. One per cent sounds survivable until you multiply it, and until you read what the harmful third of it says.

Then multiply it. On 26 October 2024 the Associated Press reported that “Over 30,000 clinicians and 40 health systems, including the Mankato Clinic in Minnesota and Children's Hospital Los Angeles, have started using a Whisper-based tool built by Nabla” — a tool used for an estimated 7 million medical visits. The same investigation documented the part that matters most here. Nabla's tool erases the original audio, which makes transcript-to-audio comparison impossible. There is no oracle to compute, no lattice to inspect, no segment to replay. The claim “the transcript is what the microphone heard” has been made unfalsifiable by the product's own retention policy. Nabla replied on 28 October 2024 that it had annotated 7,000 hours of medical-encounter audio to train its own model, and that the transcript is not directly included in the patient record.

A report limited to oracle versus selected hypothesis error would have shown none of this. In the Nabla deployment it could not even be produced. Notice what the 1% figure required: someone holding the audio and the transcript side by side. Delete the audio and the measurement goes with it. The failure becomes not merely unmeasured but unmeasurable, and every downstream reader inherits a document with nothing to check it against.

What the search could have found is one question. How hard it looked is a different one. Oracle WER answers the first and beam answers the second. Running them together is how a team concludes a decoder is doing well when it is only searching widely. Shallow fusion and hotword are the two doors through which an outside expectation enters the decision. The field defined the first for itself in a paper; the second is usually defined by whoever wrote the config file. The definition comes from a 2017 paper by Kannan and five co-authors: “Shallow fusion refers to log-linear interpolation with a separately trained language model at each step of the beam search.” On Google Voice Search, shallow fusion with a wordpiece neural LM gave a 9.1% relative word error rate reduction over their attention-based sequence-to-sequence baseline, and removed the need for second-pass rescoring. Each of the four terms below can require different proof, a different unit, and a different person to sign for it.

Context is not one feature with one setting. It helps four products for four different reasons, and each of them tolerates a different kind of mistake. Medical and legal dictation and broadcast transcription need the same mechanism as the other two, under stricter conditions. A Whisper-based tool inside 40 health systems can be a success on its own dictation slice and a failure the moment roughly 1% of its output carries sentences the audio never held. The gains are real and published, which is what makes the trade genuine rather than rhetorical. A 2020 paper on personalising an RNN-T recognizer, by Le and five co-authors, reports: “We show that these combined techniques result in 15.4%-34.5% relative Word Error Rate improvement compared to a strong RNN-T baseline which uses shallow fusion and text-to-speech augmentation.” Every row below has its own units and its own burden of proof. The case above discharges neither.

Tuning a contextual decoder safely means scoping the boost and keeping the evidence that would convict it. The exercise is finished when another team can ask which data the beam settings were tuned on and get a straight answer, and can ask to hear the audio behind a disputed line and be given it.

Four moves carry the work. Create scoped context sets, separating contacts, catalog terms, locations and temporary meeting vocabulary, so a list built for one never fires inside another. Sweep bias strength and plot entity recall against false insertions. Those two curves move in opposite directions, and a single number hides which one you bought. Test adversarial acoustics — noise, and common words confusable with the boosted entities — since a boosted name becomes the cheapest thing to write down exactly where the audio is unclear. Add lifecycle controls: expire the context, log its source, and let users remove sensitive entries.

Then three notes, following the shape of the work. First, what does defining the search space assume? Then one counterexample to that assumption: a stretch of audio where it is wrong, the noisy segment the boost should never have reached. Then what auditing search behavior triggers once that segment turns up — whether the boost is narrowed, given a shorter life, or switched off for that context entirely. Note the precondition buried in step two. You need the segment. A pipeline that deletes the original audio, as the Nabla tool does, cannot run this exercise at all.

An evaluation earns its place when it is able to contradict itself. Decoder latency, memory and partial stability will sometimes conflict with oracle versus selected hypothesis error, and that conflict is what the evaluation is for. The plainest version is a beam widened: the search looks harder, latency and memory rise, and the oracle barely moves. A bigger beam that finds nothing new is a cost with no result attached. Only a portfolio carrying both halves can show that this is what was bought.

The cleanest measurement of that gap comes from neural machine translation rather than ASR, and it should be read with that label attached. In 2019 Stahlberg and Byrne built an exact inference procedure, combining beam search with depth-first search. They used it to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set. Their finding: “Surprisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100.” Worse for anyone who assumes a wider search is a better one: for more than 50% of the sentences, the model assigns its global best score to the empty translation. Search quality and model quality are two different quantities. Widening the beam moves you toward an optimum that, half the time in that setting, is a degenerate output nobody wants.

The portfolio should also state when the team abstains or falls back. Search settings fitted to the same data used for the final report are one of those conditions. A gain measured that way is not a gain anyone can ship.