Text-to-Speech as a Product and the Text Front End

Follow one of those errors forward and you can see why the audio is incapable of reporting it. A TTS system converts text, or structured intent, into speech in three stages: a text front end, then an acoustic or generative model, then a waveform synthesizer. What the front end settles — normalization, pronunciation, language, punctuation, markup, context — is exactly what the model is then asked to say. Fluency is manufactured at the far end of that chain, at the waveform stage. It carries no information about the text that entered at the near end. Whatever the front end got wrong arrives at the listener intact, in the same assured voice as the rest.

That also means no single number can carry a release decision. How accurately the system normalizes and pronounces each type of entity is one question. Whether a listener comes away with the right content when a sentence contains names, numbers, or a switch between languages is a second question, and a different one. Neither should stand in for the other. Neither on its own is the decision.

Take the case where a silent resolution costs the most. It is not a hypothetical the lesson invented. It is a hazard class that safety bodies have already written down and named.

Two entries on the ISMP List of Error-Prone Abbreviations, Symbols, and Dose Designations turn on a single character. “Lack of a leading zero before a decimal point (e.g., .5 mg)” is meant to say 0.5 mg. It is “Mistaken as 5 mg if the decimal point is not seen”. “Trailing zero after a decimal point (e.g., 1.0 mg)” is meant to say 1 mg. It is “Mistaken as 10 mg if the decimal point is not seen”. Both are ten-fold errors. The list is the Institute for Safe Medication Practices', in its 2024 edition.

Both entries carry a double asterisk, and the asterisk has an accreditation standard behind it. ISMP's note to the table says so: “In the Table, error-prone abbreviations, symbols, and dose designations that are included on The Joint Commission’s “Do Not Use” list (Information Management standard IM.02.02.01) are identified with a double asterisk (**) and must be included on an organization’s “Do Not Use” list.” That list was announced in 2003 and banned in accredited hospitals by April 2004. Since 2010 the notations have been carried as elements of performance 2 and 3 under IM.02.02.01. Nine notations are banned in all, trailing and missing zeros among them — the count comes from the Agency for Healthcare Research and Quality's review Making Health Care Safer II.

And the ban is not confined to handwriting. These designations “should NOT be used when communicating medical information verbally, electronically, and/or in handwritten applications”, ISMP states. So a front end that meets ".5 mg", resolves it by rule, and hands one reading to the synthesizer is guessing in the spoken channel. That is the channel the ban explicitly names. And the class is one two safety bodies have already declared unresolvable by guess. Naturalness is what hides it. Whichever way the rule went, the audio comes out equally certain. Nothing in the waveform distinguishes 0.5 mg from a tenfold overdose.

An accuracy score reported per entity type does not close that gap by itself. It tells you how often numbers are read correctly across a set. One silent choice about one ambiguous dose can still pass through unnoticed.

Kestrel's transducers are one stage among several, and it pays to be able to say which stage you mean. A name can be mispronounced in a pronunciation lexicon, in grapheme-to-phoneme conversion, in SSML, or in text normalization. Four separate places, four separate owners. Collapse any two of them into a single word and you lose track of which stage owes the evidence, in what unit, and to whom. Two of the four are not loose engineering terms at all. They have a W3C Recommendation under them, with a version number and a date. The maintenance surface is specified rather than improvised.

Those places do not sit side by side. They sit in order, and each hands its output to the next. Written forms such as dates, units, and abbreviations are mapped to spoken forms by text normalization. Grapheme-to-phoneme and speech synthesis then take over further down the chain, the first producing a pronunciation representation and the second a waveform.

What the middle column costs is measurable. In 2020 a project called WikiPron mined 1.7 million pronunciations out of Wiktionary, across 165 languages, dialects and scripts. Its authors could not do that without pulling varieties apart first: “Many Wiktionary pronunciations are paired with dialectal specifications, as exemplified by Figure 1. If these specifications were simply ignored, we would obtain a large number of pronunciation variants for each word.” So English appears twice in their table: “English (UK, R.P.)” with 52,425 entries and “English (US, Gen. Am.)” with 48,556. A lexicon encoding one community's version of a word is not a caution here. It is a separate resource, with its own row and its own count.

Their generic grapheme-to-phoneme models use one architecture and one pipeline across all fifteen evaluation languages. Only early stopping and dimensionality selection are chosen per language, on the development set. Neural seq2seq word error then comes out at 2.0% on Hungarian, 5.3% on Bulgarian and 6.0% on French at one end. At the other end: 22.6% on Georgian, 28.8% on Korean, 30.9% on Burmese and 48.2% on English (UK, R.P.). Roughly a twenty-four-fold spread in error, from the same pipeline, depending on which language you happen to ship in.

A test that clears one column therefore says nothing about the next. A result in one language says little about the next either. It is also the reason “123 King Ave” is a normalization problem rather than a synthesis problem. By the time the synthesizer sees it, the number has already been decided, and decided invisibly.

So the work is to force those decisions into the open before a listener meets them. The entity list should not be invented either. The released text-normalization data set behind the numbers in the next section is “1.1 billion words of English text, and 290 million words of Russian text”, drawn from Wikipedia and verbalized by Kestrel. The semiotic classes it is scored on give the suite a ready-made spine: DATE, MEASURE, MONEY, CARDINAL, DECIMAL, FRACTION, LETTERS, DIGIT.

A linguistic test suite is where a silently resolved number or abbreviation stops being your assumption and becomes someone else's evidence. It puts “123 pages”, “123 King Ave”, “.5 mg”, 2 mA, £5 and 16 ГБ in front of the front end, in every language the product ships in. Then it records what comes back out as text. As text, because audio is exactly where such an error stops being visible.

Three things go into that record. What the speaking contract assumes: what may be said, in which language, and with which markup, agreed before synthesis. One counterexample to it. And what synthesizing and then verifying against that contract actually triggers when the counterexample turns up.

A suite like that produces numbers, and the numbers will not agree with one another. That is the finding, not a nuisance. In 2017 Sproat and Jaitly ran an attention-based RNN over an English test set of 92,416 tokens and scored 0.997 overall. Underneath that single aggregate, from the same system and the same run: DIGIT 0.865 and TIME 0.750 in English; in Russian, MEASURE 0.883, MONEY 0.842, DECIMAL 0.867, FRACTION 0.826 and TIME 0.750. Their own abstract states the consequence: “We show that while the RNN model achieves very high overall accuracies, there remain errors that would be unacceptable in a speech application like TTS.”

Look at what the residual errors actually are. 2 mA read as “2 megaamperes” instead of “2 milliamperes”. £5 read as “five” instead of “five pounds”. 11 billion AED read as “eleven billion danish” instead of “eleven billion dirhams”. 16 ГБ read as “sixteen hertz” instead of “sixteen gigabytes”. Each is a unit or currency substitution. Each comes out fluent. Each disappears inside 0.997.

The unit matters as much as the list. A dose read two ways is an entity, not a sentence. A sentence-level score absorbs it the way an aggregate of 0.997 absorbs TIME at 0.750. Per-entity accuracy will also pull against latency, failure fallback, and provenance coverage at some point. An evaluation is worth its cost only if it makes that disagreement visible instead of averaging it away. So the portfolio has to state, in advance, what happens when an ambiguity cannot be resolved at all — where an unresolvable number or abbreviation obliges the system to abstain or fall back rather than guess.