Speech and audio
Automatic Speech Recognition as a Product System
Frame ASR as a complete product system with acoustic capture, segmentation, decoding, text policy, uncertainty, and downstream use.
By the end you can
- Define automatic speech recognition as a product system as an operational problem with explicit inputs, outputs, and boundaries
- Distinguish verbatim transcription, normalized transcript, and semantic command without treating them as interchangeable
- Trace the workflow from define transcript purpose through close the interaction loop
- Evaluate automatic speech recognition as a product system using wer and character error by language, device, noise, and speaker slice and evidence from difficult deployment slices
Example
Seven words in a hundred were wrong and the notes were signed anyway
Seven words in every hundred were wrong. The raw speech-recognition text in 217 clinical documents carried a 7.4% error rate. 96.3% of those notes contained at least one error. The documents were dictated in 2016 with speech-recognition assistance, and a 2018 study in JAMA Network Open counted the errors at each stage of the pipeline. That is the honest headline: more than nine words in ten were right, and almost every document was still wrong somewhere.
The part an aggregate rate cannot report is which words. Of the errors at the speech-recognition stage, 15.8% involved clinical information and 5.7% were clinically significant. A word error rate averages a drug name into a corpus of articles and prepositions. It returns one number in which both weigh the same. Nothing in that number separates a dropped "the" from a changed dose. That is why this lesson keeps coming back to the study. The failure is not a transcription failure. It is a product failure, and the rest of the lesson is about where in the product it lives.
- The decision being made there was not which recognizer to buy. It was whether to treat the whole chain at once: acoustic capture, segmentation, decoding, the text policy that formats what comes out, the uncertainty attached to it, and the downstream use all of it exists to serve.
- The failure is the ordinary one, not an exotic one. An aggregate word error rate of 7.4% was reported, and names, numbers and clinically significant terms were left to fend for themselves inside it.
- A team in that position can show word error and character error broken out by language, device, noise and speaker slice. That is real evidence, and usually the only evidence anyone asks for. The 15.8% and the 5.7% are a different measurement, taken on purpose.
- What nobody writes down is the sentence that would have caught this before the documents were signed: who reads or consumes this transcript, and what decision follows once they do.
Comparison
Three transcripts of the same utterance
Answer that last question and it turns out you are not building one transcript. You are choosing among three, and a dictated dose travels differently through each.
Verbatim transcription preserves disfluencies and spoken form. It serves evidence or analysis, where the hesitation before the number may be the point. A normalized transcript throws some of that away deliberately, in exchange for something a person can read and sign. A semantic command throws away almost everything and keeps the instruction — the quantity and the act. That is the only form a machine can execute.
These are not three grades of quality, with verbatim at the top. Each discards on purpose what the one before it kept. Each therefore has to be shown to work on its own terms. The 7.4% figure was measured on one of these three. It does not transfer to the others.
Verbatim transcription
Preserves disfluencies and spoken form for evidence or analysis.
- Decision focus: Define transcript purpose
- Useful evidence: WER and character error by language, device, noise, and speaker slice
- Watch for: Optimizing aggregate WER while failing names, numbers, or commands
- Best used when its assumptions are documented for automatic speech recognition as a product system
Normalized transcript
Formats numbers, dates, punctuation, and casing for readability or machines.
- Decision focus: Specify output policy
- Useful evidence: Entity, number, command, and keyword error rates
- Watch for: Language-model bias replacing rare words with common phrases
- Best used when its assumptions are documented for automatic speech recognition as a product system
Semantic command
Maps speech to an intent or structured action and requires authorization controls.
- Decision focus: Select architecture and decoding
- Useful evidence: Endpoint latency, real-time factor, and correction burden
- Watch for: Silent normalization changing legal or clinical meaning
- Best used when its assumptions are documented for automatic speech recognition as a product system
A transcript is not authorization
One operation sits under all three. Automatic speech recognition maps acoustic evidence to a symbolic sequence. A vocabulary decides which words exist. A normalization policy decides how they are written. A decoding procedure decides which sequence wins. That is what people mean when they say the model is accurate.
Whether the product is reliable rests on a longer list: segmentation, language context, timestamps, confidence, entity handling, what users are left to correct, and what happens downstream when a machine acts on the words instead of reading them. The dictation system in that clinical study was accurate in the first sense, at 7.4% error. It was unreliable in the second, with 15.8% of its errors touching clinical information. There is no contradiction in that. A transcript is a probabilistic guess about a sound. It does not prove that the speaker intended those words, that the speaker was authorized to say them, or that what was said was true.
So aggregate transcription accuracy, broken down by language, device, noise and speaker, answers one question. How often the system gets an entity, a number, a command or a keyword right is a different question with a different answer. Neither can stand in for the release decision. Shipping means answering both.
Acting on a transcript without a separate check turns a probabilistic guess into an authorized instruction.
Case
680,000 hours, thirty seconds at a time
That release decision is usually made about a model somebody else trained. Whisper is the system most product teams start from, and its published shape explains much of the behaviour they inherit with it. It was trained on 680,000 hours of audio. Of that, 117,000 hours cover 96 other languages and 125,000 hours are X-to-English translation data. OpenAI described all of this in 2022.
The window the model reads through is thirty seconds. The Whisper paper says so in its section on data processing: "We break audio files into 30-second segments paired with the subset of the transcript that occurs within that time segment." Long-form transcription is a separate engineering problem because of that sentence, not as a matter of opinion. The fixed 30-second segment is an architectural constraint the product inherits. Everything longer has to be stitched across it.
Independent benchmarking keeps those tracks separate too. The Open ASR Leaderboard benchmarks 86 open-source and proprietary systems across 12 datasets, and its 2025 abstract states the method plainly: "We standardize word error rate (WER) and inverse real-time factor (RTFx) evaluation for consistent accuracy-efficiency comparisons across model architectures and toolkits (e.g., ESPNet, NeMo, SpeechBrain, Transformers)."
In its long-form English table, NVIDIA Parakeet CTC 1.1B records an RTFx of 2793.75 at 6.68% average WER. OpenAI Whisper large-v3 records an RTFx of 68.56 at 6.43%. That is 40.7 times the throughput for 0.25 percentage points more error, and only on the long-form track. One is far faster. The other is slightly more accurate. Neither line says anything at all about what either model does with a dose or a drug name.
Figure
Visual
Decide what the transcript is for
The model settles none of that, which is why the order of the product work matters as much as the work. Three things happen in sequence. A team defines what the transcript is for — evidence, reading, or an instruction to act on. It then specifies the output policy that follows from that purpose: what is normalized, what stays verbatim, how numbers and names are written. Finally it closes the interaction loop, where a user confirms, corrects, or is stopped before the words take effect.
Everything the output policy does rests on what the purpose settled. The interaction loop is where a team learns whether that settlement held. The trouble is the direction of travel. The purpose is fixed first, by whoever is earliest and least informed. Nobody standing at the far end is asked to re-check whether it was the right purpose. The dictation workflow in the clinical study did close its loop — transcriptionist editing, then physician sign-off — and that loop was load-bearing rather than decorative. Displaying is not confirming. Only a step that can change the text is.
- 01
Define transcript purpose
Choose verbatim record, readable notes, command parsing, search index, captioning, or another use.
- 02
Specify output policy
Set language, units, punctuation, casing, numbers, names, timestamps, and unknown handling.
- 03
Select architecture and decoding
Match offline or streaming constraints, vocabulary, context, and compute.
- 04
Close the interaction loop
Add confidence, confirmation, correction, fallback, logging, and monitoring.
A transcript purpose fixed at the start shapes the interaction loop later, and nobody there re-checks whether the purpose was right.
Key idea
Aggregate WER hides the words that matter
There are four ways a transcription product goes wrong while its headline accuracy number stays healthy. Each of the four contradicts the broad claim that the product is working. None of them shows up in that number. Read them as the checks the number does not cover.
The first is the clinical case itself: an aggregate word error rate reported while names, numbers or commands go on failing inside it. The second is language-model bias, where a rare word is replaced by the common phrase that sounds like it. The third is silent normalization changing legal or clinical meaning. The fourth is executing an action straight from a transcript, with no confirmation and no policy check standing in between.
The mechanism has been measured. Koenecke and colleagues put a number on it in a 2024 study of Whisper called Careless Whisper: "While many of Whisper's transcriptions were highly accurate, we find that roughly 1% of audio transcriptions contained entire hallucinated phrases or sentences which did not exist in any form in the underlying audio." Of those hallucinations, 38% carried explicit harms. They fell disproportionately on speakers with longer non-vocal durations, a symptom of aphasia. Text that no speaker ever uttered, punctuated and formatted like the rest, concentrated on the slice least able to object.
None of these is a transcription error in the ordinary sense, and most of them produce text a careful reader would accept. That is what the four have in common. Punctuation, formatting and language-model repair make the output look more confident than the acoustics ever supported.
A headline accuracy number can hold steady while every name, number and command in the transcript goes wrong. The clean punctuation around them makes the error harder to catch.
Analogy
A stenographer plus an editor plus an operator
It helps to count how many jobs are being done at once. Three sit between the sound and the action. A stenographer writes the words down. An editor formats them into something readable. An operator acts on the result. Each can introduce a different kind of error, and three people leave three places to look.
The clinical study is unusual because it measured those three stations on the same 217 documents. The raw speech-recognition text carried a 7.4% error rate. After professional transcriptionist editing the rate was 0.4%. After physician sign-off it was 0.3%. The editing pass is worth almost the whole drop. That is the seam made visible: the stenographer's output and the signed document are not the same artifact, and only one of them was ever fit to act on. Zhou and colleagues put the conclusion in those terms in JAMA Network Open in 2018: "Seven in 100 words in SR-generated documents contain errors; many errors involve clinical information. That most errors are corrected before notes are signed demonstrates the importance of manual review, quality assurance, and auditing."
Modern ASR performs all three functions jointly, inside learned models and decoders. Nothing separates them at runtime and nothing separates them in the output, so the errors arrive already mixed together. The seams have to be reintroduced deliberately, in the evaluation and in the workflow. The system will not hand them to you. 7.4% against 0.4% is the size of what is at stake.
Evaluate the transcript policy and downstream action, not only acoustic word substitutions.
Steps
Draft an ASR output contract
Reintroducing the seams starts with writing them down, and the document is short. An ASR output contract records three things. What you assume the transcript is for. One counterexample to that assumption — a dictated dose, if clinical detail is what your users say out loud. And what has to happen once the interaction loop closes and the words are acted on. Numbers, names and commands are where such a contract earns its keep. They are exactly where a good average score is least informative.
The most useful model of the genre was written by a regulator. The FCC's caption quality standards require captioning to be accurate, synchronous, complete and appropriately placed — four named dimensions, each with its own text. The accuracy clause is an output policy rather than an aspiration: "Captioning shall match the spoken words (or song lyrics when provided on the audio track) in their original language (English or Spanish), in the order spoken, without substituting words for proper names and places, and without paraphrasing, except to the extent that paraphrasing is necessary to resolve any time constraints." Proper names must not be substituted. Numbers must be represented accurately. Paraphrase is allowed only against a stated constraint, and the exception is named. That is what a clause looks like when someone expects it to be enforced.
Write yours so that someone else can use it against you. If another team cannot pick it up and turn it into a challenge to a release resting on a strong average score, the contract is not doing its job. That release will pass.
1. Choose one use
State who reads or consumes the transcript and what decision follows.
2. Define text transformations
List allowed normalization, punctuation, redaction, and unknown markers.
3. Identify critical tokens
Name entities whose errors carry disproportionate cost.
4. Design confirmation
Specify when the system must ask, display, repeat, or refuse.
Write down where the system must ask, display, repeat or refuse before words are acted on. The interaction loop closes whether or not anyone specified it.
Example
Names, numbers, commands: their own error rates
A contract is only as good as the numbers that can be held up against it, and no single number can do that job. Four kinds of evidence belong in the report. The reason for keeping all four is that they fail in different places: any one of them can read healthy while the product is broken.
The slice that shows this most sharply is racial disparity. Five commercial ASR systems — Amazon, Apple, Google, IBM and Microsoft — were tested on 19.8 hours of interviews with 42 white and 73 black speakers. The abstract reports what came back: "We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers." Koenecke and colleagues published that in PNAS in 2020.
0.35 against 0.19, in one corpus, across five vendors at once. A single corpus-wide figure sits somewhere between those two numbers and describes neither group. That is what "report WER by speaker slice" means in practice. It is also why the first kind of evidence below has to be printed as a table rather than as a headline.
- The core task evidence is word error and character error, reported by language, device, noise and speaker slice rather than as one headline figure — the difference between 0.35 and 0.19 exists only if someone splits the corpus.
- The system behaviour to measure separately is entity, number, command and keyword error rates: the words that carry consequences, the 15.8% inside the 7.4%.
- The robustness slice is endpoint latency, real-time factor and correction burden — whether the system keeps up, and how much of the work it hands back to the user, as the clinical transcriptionists were handed it.
- The lifecycle evidence is high-impact action errors, abstention quality and user recovery: what happens after a wrong word has already been acted on.
Word error and character error by language, device, noise and speaker slice mean little unless high-impact action errors, abstention quality and user recovery are printed beside them.
Key takeaways
- A transcript is a probabilistic guess about a sound, so treating it as authorization is a decision someone has to take deliberately — it is never a property the recognizer supplies.
- Accuracy and reliability are different results: the dictation systems in the 2018 JAMA Network Open study carried a 7.4% error rate and still put clinical information wrong in 15.8% of their errors.
- Decide what the transcript is for before anything else, because the output policy inherits that decision and the interaction loop is the only place anyone finds out it was wrong.
- Verbatim transcription, a normalized transcript and a semantic command each discard on purpose what the others keep, so each has to be shown to work on its own terms.
- Names, numbers and commands need their own error rates: roughly 1% of Whisper transcriptions in a 2024 study contained entire hallucinated phrases absent from the audio, 38% of them carrying explicit harms, in text a careful reader would accept.
- Report word error and character error by language, device, noise and speaker slice alongside high-impact action errors, abstention quality and user recovery — a corpus-wide average that hides 0.35 against 0.19 will look fine on its own.