ASR Evaluation Beyond Word Error Rate

Word error rate is not folklore. It has a written definition, and the body that scores the runs is the one that wrote it. NIST's OpenASR20 evaluation plan, published in 2020, gives the formula as (#Deletions + #Insertions + #Substitutions) / #ReferenceWords. It also names the software that computes it: “The primary metric computed on the submitted output is Word Error Rate (WER), as implemented in the sclite tool of the Speech Recognition Scoring Toolkit SCTK available from NIST.” Three edit types, one denominator, one named tool.

Read the same plan a little further and a second property appears. Before any of that arithmetic runs, the reference itself is rewritten: “Prior to scoring, normalization is performed so the WER obtained is as precise as possible.” The number does not exist until somebody has decided what counts as a word. Tokenization, normalization, reference policy and alignment all have to be settled first. An agency running a ten-language challenge settles them in writing. Most teams settle them by habit.

The formula also gives every word the same weight. A dropped article and a wrong drug name each add one to the numerator. Equal weighting and pre-scoring normalization, between them, hide most of what a released system is actually judged on: semantic harm, entity correctness, punctuation, speaker attribution, latency, calibration, and the editing burden left to a human. WER is unstable on short utterances too. And two figures produced under different normalization policies cannot be compared, however similar they look.

The answer is not a cleverer single number. It is a written evaluation contract of the kind NIST publishes: what is being measured, under which policy, carried through to the release decision it is meant to inform. With a named owner at each end.

Equal weighting has a price, and a clinical pipeline shows where it is paid. A stratified random sample of 217 clinical notes was followed through that pipeline stage by stage. The notes were dictated by 144 physicians in 2016, at two health care organizations using Dragon Medical 360 | eScription. JAMA Network Open published the result in 2018: “Among 217 clinical notes randomly selected from 2 health care organizations, the error rate was 7.4% in the version generated by speech recognition software, 0.4% after transcriptionist review, and 0.3% in the final version signed by physicians.”

That one sentence contains the whole argument for measuring human effort. The raw recognizer output was wrong at 7.4 per cent. What turned that into 0.4 per cent was a transcriptionist. What turned it into 0.3 per cent was a physician reading the note before signing it. Neither of those two people appears anywhere in an aggregate error rate. Both of them were paid.

The severity split is the other half. Of the errors at the speech-recognition stage, 5.7 per cent were clinically significant. An edit count cannot separate that 5.7 per cent from the rest. In the arithmetic they weigh exactly the same as every other error. So a release review that asks only for aggregate error rates cannot tell a transcript that is cheap to fix from one that is dangerous to miss. Everything below is an attempt to build a review that can.

A very small vocabulary sits under all that policy. The NIST formula counts three edits and no more: deletions, insertions and substitutions, over reference words. The thing that made 5.7 per cent of the clinical errors clinically significant is the one it does not count. These four words look interchangeable in a report. They are not. Each implies different evidence, different units and different decision rights, so an evaluation write-up has to use them exactly.

Those three counted edits are all WER has: general sequence-edit error at the word level, and then it stops. An entity or semantic metric asks a second question of the same transcript. A product outcome asks a third. That they are genuinely different claims is not an assertion in this lesson. It has been measured, with WER pinned in place.

Three hypothesis sets were built with an identical 7.44 per cent WER on a 23k-utterance open-domain test set. Kim and colleagues published them in 2021. Named-entity F1 across those three sets ranged from 0.590 to 0.846 — a spread of 0.256, with the edit count held constant to two decimal places. On a 10k-utterance assistant set, again at an identical 6.16 per cent WER, intent accuracy ranged from 94.28 to 96.22 per cent. Their own summary of the effect is blunt: “We observed that as SemDist reduces entity F1-score increases, even with the same WER (Set A vs. Set C).”

So the three columns below are not three views of one quantity. A team that reports 7.44 per cent has said nothing about whether the entities came out right. A team that picks a system on that basis has, in that setup, a range of a quarter of an F1 point to be wrong inside. Each column has to be evidenced on its own terms rather than inferred from the edit count.

WER is general sequence-edit error at the word level. It is what the evaluation contract settles first, and it is read off WER, CER and edit-operation profiles. Systems scored under different normalization cannot be set beside one another on it.

An entity or semantic metric weights domain-critical content and meaning more directly. The work it drives is building representative slices. Entity, number, keyword and command error are where it shows, and one average published without uncertainty or slice coverage hides every one of them.

A product outcome measures task completion, editing time, user recovery or downstream harm. The 0.4 per cent after transcriptionist review is one of these, not a WER. It is the whole reason to add task metrics, and it lives in latency, stability, calibration and abstention. It goes wrong the moment an ASR model helps create its own test references. Each of the three is best used when its assumptions are documented.

Evaluation programs have three moves in them, and teams reliably skip the middle one. Defining the evaluation contract comes first and connecting it to release decisions comes last. Between them sits creating representative slices. A plan that jumps the gap arrives at the release meeting having tested nothing in between.

The cost of skipping it has been measured across an industry at once. Five commercial ASR systems — Amazon, Apple, Google, IBM, Microsoft — were scored on 19.8 hours of interviews with 42 white and 73 black speakers across five US cities. Ten researchers ran the study, and PNAS published the result in 2020: “We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers.”

A 0.16-point gap between two speaker populations, on every one of five systems, sat inside numbers each vendor could report as a single average. The gap held on identical phrases. That is what let the authors trace it to the acoustic models rather than to what the speakers were saying. No amount of care in step 1 or step 4 surfaces that. Only a slice named in advance does. That is where a plan earns its keep.

The four steps: 1. Define the evaluation contract — fix transcript style, normalization, segmentation, reference policy and scoring tools, as NIST does in a published plan. 2. Create representative slices — cover language, accent, device, noise, domain, age, rate, overlap and duration. The racial-disparity study is what a missing slice looks like once someone measures it; OpenASR20's ten low-resource languages are what a missing language looks like. 3. Add task metrics — measure entities, numbers, commands, speaker attribution, latency and correction effort. 4. Connect to release decisions — set thresholds, confidence intervals, failure budgets and rollback conditions.

The moment to prepare for is the one the clinical pipeline makes visible. The recognizer's own output was wrong at 7.4 per cent. The signed note was wrong at 0.3 per cent. The entire distance between those two figures is human labour that no edit metric reports. So when WER, CER and edit-operation profiles point one way and human correction time, task completion and severe-error review point the other, the plan has to say in advance which side the release decision listens to. It should also say what happens when two systems turn out to have been scored under different normalization. The comparison is withheld, not published as a ranking.

An evaluation plan is finished when a rival team can read it and find no way to dismiss the comparison it reports. No unnamed corpus, no undocumented normalization, no average standing alone. NIST publishes such a document for a public challenge, down to the formula and the scoring tool. A team shipping into a hospital or a call centre has less excuse for keeping its policy oral, not more.

Write down three things: what your evaluation contract assumes, one concrete case that breaks it, and what the release decision does when that case appears.

The four steps in order: 1. Freeze the text policy — document casing, punctuation, numbers, hesitations and partial words. On some datasets normalization alone moved the reported figure by up to 50 per cent. 2. Define critical slices — allocate enough examples for high-risk populations and acoustic conditions; 42 and 73 speakers across five cities were enough to expose a 0.16-point gap across five vendors. 3. Add human workflow measures — time corrections and record the kinds of edits required. That is the only way the distance from 7.4 to 0.3 per cent shows up as a cost. 4. Set decision rules — use confidence intervals and explicit release or stop criteria.