Multilingual, Low-Resource, Accented, and Code-Switched ASR

Five commercial speech recognizers were put through the same audio at once: Amazon, Apple, Google, IBM and Microsoft. The material was 19.8 hours of matched interview recordings from 42 white and 73 black speakers. The result was not a wobble in one vendor's product. Koenecke and colleagues published it in PNAS in 2020: “We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers.”

Read that pair of numbers slowly. Almost every argument in this lesson is in it. Each of those five systems could report a headline error rate it was proud of. None of those five figures would contain the sentence above. An average over both groups is the operation that dissolves a 0.35-against-0.19 split into one number belonging to nobody.

A per-group table is the minimum that makes the gap visible. A table is still only a display. It shows that a group of speakers is being transcribed worse; it does not say why. And the cheapest explanation is sitting there for any team to pick up: those speakers are pronouncing the language wrongly. The PNAS authors closed that distance. The rest of this lesson is about how they closed it, and about who carries the cost when nobody does.

“One shared model” is an architecture choice. It is worth seeing exactly what a team buys with it. A universal shared model maximizes transfer and operational simplicity: one thing to train, one thing to serve, every language able to borrow structure from every other. The bill arrives in the same breath. Languages and varieties then compete for capacity inside it. The ones with the least training audio lose that competition silently, in deletions rather than in complaints. Nothing in a 0.19 average announces that some group inside the mixture is at 0.35.

There are two other ways to divide the same capacity. A shared encoder with language-specific heads keeps the common structure and gives each language somewhere of its own for whatever does not generalize. Separate or adapted models give up most of the sharing in order to end the competition altogether.

None of the three is right in general. A good result from one settles nothing about the other two. Each has to be defended language by language, and variety by variety. That is heavier than it sounds. The case for the shared model has to be made again, on its own, for the speakers who were losing.

Competition for capacity is not the only thing deciding who benefits. Multilingual ASR can share acoustic and linguistic structure across languages, and that sharing is real. But capacity, tokenizer design, sampling, script, domain, and language identification determine who benefits. Not one of those six is set by how many languages appear on the list.

The word to be careful with is accent. The PNAS study is why the care is not optional. Its authors did not stop at the 0.35-against-0.19 gap. They compared identical phrases spoken by both groups, and the gap was equally large there. That removes the explanation that black speakers were saying different, harder, or rarer words. Same words, same size of error gap. On that evidence the authors trace the disparity to the acoustic models rather than to the speakers.

That is what it looks like to close the distance between a gap and its cause. It is also why accents and dialects have to be treated as ordinary ways that language varies, not as noise to be normalized away. A system built to normalize variation has already decided that one way of speaking is the language and the rest are departures from it. The error rate it then reports will read like a fact about the speakers. It is a fact about the system.

There is an operational boundary here too, and it is the one crossed in procurement. A broad language list does not demonstrate equal quality, culturally valid text normalization, or support for code-switching. What a vendor reports as coverage may leave out how speech varies by dialect, age, device and domain. Those are precisely the columns a per-language table does not have either. Five vendors could each have claimed English. The disparity was in all five.

Defending against this is a question of ownership rather than technique. A team decides which communities and which uses the system is for. It carries that decision through to evaluating with native speakers of each language. Then it names who owns each end: who chose the communities, and who signs off that the evaluation actually reached them.

Evaluation is the binding constraint, and the path below sets out where it comes from. Communities, tokenizers, models, and native evaluation are four separate decisions in a system that must handle many languages, thin data, accents, and mid-sentence switches. The path refuses to average them into one score. Averaging them is exactly the operation that turns 0.35 and 0.19 into a single figure a team can ship on.

The order matters more than four boxes suggest. The tokenizer is decided early, close to the data, by people thinking about sequence length and compression. Native evaluation is decided last, close to launch, by people thinking about whether to ship. In between, nothing goes back.

That is how a tokenizer that fragments one language's words reaches production untouched, and arrives at the end of the path dressed as a property of the language. It is also why the identical-phrases comparison in the PNAS study had to be run deliberately. Nothing in the pipeline produces it on its own.

That argument turns on four words, and they are not the same kind of thing as each other.

Token fertility is a count: how many pieces the tokenizer needed. The size that count can reach has been measured. It is not hypothetical. In 2023 Petrov and colleagues put it in one line: “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.” They also show that even character- and byte-level models leave over 4 times the encoding-length difference for some language pairs. A change of granularity does not quietly remove the effect.

Negative transfer is a loss — the price of shared capacity, which the three architectures divide up differently. Code-switching is neither a count nor a loss. It is ordinary behavior, and both of the others mishandle it. Underneath all three sits a language tag, which is not a measurement at all but an assumption: that there is one language in this audio, and the system already knows which one.

Which is why the title of this section is a claim rather than a definition. Calling code-switching an error rather than behavior changes what has to be measured, in what unit, and who is allowed to call the system ready.

An evaluation worth running is one that can produce a sentence like “0.35 for black speakers compared with 0.19 for white speakers” before a launch rather than after it. WER or CER broken out by language, dialect, script and domain is the base of that report. Tokenizer fertility belongs beside it, because fertility is the number that explains a gap the per-language table can only display. Rare-entity error belongs there for the same reason. The headline figure averages the worst-served speakers away — that is what an average is for. On its own it will not tell you that a dialect is being scored as a pronunciation error.

The Artie Bias Corpus is a small, concrete instrument built for exactly this: 1,712 expert-validated clips, about 2.4 hours, drawn from Mozilla Common Voice. Meyer and colleagues were blunt about what they found in 2020: “We observe a significant accent bias in our baseline DeepSpeech model, with more accurate transcriptions of US English compared to Indian English.” The numbers behind it are a baseline character error rate of 40.50% on Indian English against 21.50% on US English, at p = 8.54e-32. A gender gap of 2.74% CER was not significant.

Then the part worth pinning to the wall. After fine-tuning on all of Common Voice, they inadvertently created a significant gender gap: 3.55% CER, p = 0.042. Improving the system introduced a disparity that had not been there. Only a matrix with a row for it could have caught that. No averaged headline number ever would.

Every figure also needs four things attached before it means anything. Name the unit it is in. Say which speakers it was measured on. Say how uncertain it is. Say where and how the audio was recorded. That last one is not bookkeeping. 19.8 hours of interview audio and roughly 12 hours a language of read Wikipedia sentences are not the same measurement. The device and the room are part of what was measured.

The exercise is the smallest version of all of this. Build a multilingual evaluation matrix, and judge it by one property: it names the languages and the speaker groups the system is not ready for. A matrix that lists only what works is a language list with extra columns.

It is written for someone else to use. Another team should be able to pick it up and push back when a dialect is being written off as a pronunciation error. They should be able to notice a 3.55% gap that appeared because of a fine-tune rather than in spite of it. Give the matrix the rows that make that possible: a group per row, a fertility column, and the recording conditions written down beside each figure.

Three notes are enough. Write down what your mapping of communities and use assumes — which speakers you decided the system is for. Then write one counterexample to it: a speaker or a use the mapping quietly leaves outside. Then write what evaluating with native speakers changes once you have their result. That is the step that either confirms the map or shows you what it was hiding.