Acoustic Units, Phonemes, Characters, and Subwords

Two things are fixed the moment you choose the output units of an ASR system: the sequence the model learns, and how finely it can handle a form it has never seen. Machine translation stated the problem first, and most compactly. A 2016 paper opens with the sentence the whole subject turns on: “Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem.” Sennrich and two colleagues wrote it. Their fix was to build words out of smaller learned pieces — byte-pair encoding. Three kinds of word drove it: names, handled by character copying or transliteration; compounds, handled by compositional translation; and cognates and loanwords. Against a back-off dictionary baseline, their subword models gained up to 1.1 BLEU on the WMT 15 English-German task and up to 1.3 BLEU on English-Russian. That is what "subword units arrived from machine translation in 2016" actually amounts to. A named baseline, two language pairs, and gains of that size.

There is no single inventory to reach for. The choice changes what the model can even be wrong about. CMUdict ships 39 phones and 135,166 entries for English. The CTC experiments on TIMIT worked over a different inventory again, and the 2006 paper prints its exact size: “The input layer was size 26, the softmax output layer size 62 (61 phoneme categories plus the blank label), and the total number of weights was 114,662.” The blank is a real output unit, not a notational aside. The BLSTM/HMM hybrid they compared against had 61 units instead of 62 precisely because it carried no blank label. With prefix-search decoding, CTC reached a label error rate of 30.51% ± 0.19. The context-dependent HMM reached 35.21%, the hybrid 33.84% ± 0.06.

Three inventories — 39 phones, 61 phoneme categories plus a blank, a learned subword vocabulary — and three different things being counted. Each one moves the rare-word problem rather than removing it. Phones can encode pronunciation structure. Characters and bytes give an open vocabulary. Subwords trade sequence length against pieces that can be reused. None of the three solves the ways languages vary. Orthography, pronunciation, script, morphology, code-switching, text normalization and training coverage go on deciding whether a sequence is learnable, whatever inventory sits on the output side. Which is why the choice starts earlier than the architecture.

Earlier means starting from the languages rather than the model. A writing system, a sound system, the modeling choice and the evaluation of rare forms are four separate decisions. The path below keeps them in that order instead of collapsing them into one score.

The order is what protects you. Whatever mapping you assume between the marks on the page and the sounds underneath is assumed before any unit is chosen, and rare-form evaluation then runs on top of it. One language can be mapped two ways and give you two different systems. That is what Sainath and colleagues did in 2018. Their grapheme systems were built on 74 symbols. Their phoneme systems used 45 context-independent phonemes for US English, and a unified set of 50 for the multi-dialect model. Those counts are the first decision, not the last, and every later number is quoted relative to them. Get the mapping wrong and the evaluation does not come back saying so. It comes back as a number, a little worse than the last number, with nothing attached to say which of the four decisions produced it.

The four steps, in order: map the writing and sound systems, documenting script, pronunciation variation, word boundaries, morphology and borrowed forms; choose candidate units, comparing phones, graphemes, characters, bytes and learned subwords; design the lexicon or tokenizer, controlling unknowns, reserved symbols, normalization and multilingual sharing; then evaluate rare forms — names, compounds, dialects, code-switching, unseen vocabulary.

The clearest published version of that collapse is the one the 2016 subword paper set out to fix. A fixed vocabulary handles frequent forms accurately and has nothing to write for the rest. The words a system most needs to get right — names, compounds, cognates and loanwords — are precisely the words a closed vocabulary cannot spell. The remedy was to stop treating the word as the smallest unit. Rare and unknown words were segmented instead into pieces learned by byte-pair encoding. The comparison reported is against a back-off dictionary baseline, on the WMT 15 English-German and English-Russian tasks, with gains of up to 1.1 and 1.3 BLEU.

That is a modest-looking pair of numbers for a change that alters what the system can represent at all. It is also the shape of the trade every ASR team meets. Smaller units improve coverage. They also lengthen the output sequence, and they leave pronunciation ambiguity untouched. Smaller pieces let a system write down a name it was never trained on. They do not tell it which name it heard. Measured only by tokens per second and sequence length, the switch reads as a straight cost increase. Neither of those two numbers records what was bought.

Fixed vocabulary, open vocabulary, subword, phoneme — the 2016 framing is easy to repeat loosely, and loose repetition is where this subject goes wrong. A grapheme and a phoneme are not two words for one unit. One belongs to a script, the other to a sound system. The 2018 comparison keeps them apart by counting them separately: 74 grapheme symbols against 45 context-independent phonemes for US English. Subword and open vocabulary are not two words for one idea either. They describe what a model can still represent when it meets a word it has never seen. Treat any two of them as one and three things shift at once — the proof you owe, the unit you measure it in, and the person who gets to decide.

One published comparison sits under one set of pressures. Change the product and the same six-way comparison pulls in a different direction. None of the four cases below inherits another team's answer, or the proof behind it.

Fragmentation is the second of those four and the one that hides best, because it does not spread evenly. It has also been measured. One text, translated into several languages, does not come out the same length in tokens. A 2023 study of language model tokenizers reports the spread: “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.” The obvious escape does not work. Even character-level and byte-level models still showed over 4 times the difference in encoding length for some language pairs. Dropping to smaller units narrows the disparity. It does not remove it.

An average taken over a whole corpus is the wrong instrument for a cost that lands on one language, one dialect or one domain. The sharpest demonstration of that is a dialect slice, not a language one. Five commercial ASR systems — Amazon, Apple, Google, IBM, Microsoft — were tested on 19.8 hours of audio from 42 white and 73 black speakers: 2,141 matched snippets per group, 4,282 in total, averaging 17 seconds each. Koenecke and colleagues published the result in PNAS in 2020: “We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers.” Five deployed systems, matched audio, and a gap that no headline corpus average would have shown.

So report the numbers with their provenance attached. Spell out the unit, the corpus it was counted over, how uncertain the count is, and the decoding setup it was taken under, with unknown or fallback frequency beside it.

The document that would have caught these failures before launch is a short one. An ASR vocabulary audit holds data and model budget still and moves the units alone. That is what makes the 2018 dialect table readable at all: matched architectures, one variable changed, 74 grapheme symbols against 45 or 50 phonemes, and the WER difference attributable to that. The audit is written to be used against you. Another team should be able to pick it up and challenge a vocabulary fitted on text it should never have seen. Three things go into it: what your mapping between writing and sound assumes, one counterexample that would break that assumption — a proper noun of the "Reese" and "Rhys" kind, if names are what you sell — and what a check on rare forms triggers when it fails.

Four steps. Collect hard text: names, numbers, compounds, code-switches, spelling variants. Tokenize and count, measuring fragmentation, unknowns and sequence expansion. Inspect the pronunciation assumptions, comparing written and spoken forms across communities. Then run a unit ablation, holding data and model budget stable while changing only the output unit.