Text Normalization, Punctuation, Timestamps, and Confidence

Dictated clinical notes carry 7.4 errors per 100 words. Zhou and colleagues counted them: 217 notes dictated in 2016 at two US health systems, published in JAMA Network Open in 2018. Their finding, in their own words: “The error rate in SR notes was 7.4% (ie, 7.4 errors per 100 words). It decreased to 0.4% after transcriptionist review and 0.3% in SNs.”

96.3% of the speech-recognition notes contained at least one error. 5.7% of the errors at that stage were judged clinically significant. Read the three numbers as a sequence: 7.4%, then 0.4%, then 0.3%. What removed the errors was a transcriptionist reading the note and a physician signing it. No formatting score sits anywhere in that chain.

That is the gap this lesson is about. A polished line does not announce which of its properties came from the speaker and which came from a rule. Entity-level normalization accuracy by type is the one measurement usually watching this stage. It asks whether a transformation applied cleanly. It never asks what the speaker meant. Everything a system does to a transcript after recognition is harder to check than recognition itself, precisely because its output arrives looking finished.

It is worth naming that layer precisely. ASR text post-processing maps spoken-form tokens into written conventions: punctuation, casing, entities, timestamps, and redactions. These transformations are useful, and nobody wants to read raw spoken form. They are also ambiguous, domain-dependent, and harder to audit than the raw recognized tokens they replace. A misrecognized word looks wrong. A wrongly formatted one looks finished.

The ambiguity is not a corner case. It is the textbook definition of the task. A 2019 paper in Computational Linguistics sets it out with a worked example: “In this application, one must decide, for example, that 123 is verbalized as one hundred twenty three in 123 pages but as one twenty three in 123 King Ave.” The same three digits, two correct readings, and only the context decides. The same paper reports that neural normalizers occasionally emit what its authors call unrecoverable errors — reading 3 cm as three kilometers — and that finite-state covering grammars suppress them. Unrecoverable is the operative word. Nothing downstream can undo it, because the evidence it contradicted is no longer in the text.

So the first thing to settle is which document is being produced at all. Spoken form, written normalization and structured extraction each hand the reader a different document. A team that has not chosen between them will ship all three by accident.

Confidence belongs to this same layer and inherits the same difficulty. It is not one universal probability. Token scores, sequence scores, calibration models, agreement signals and timestamps depend on the decoder, the normalization, the segmentation and the target event being predicted. Two numbers labelled confidence in the same product may not be answering the same question.

The choice deferred a moment ago is this one. Spoken form keeps words close to what was uttered, disfluency and verbalized numbers included. The digits stay as one twenty three, the ambiguity intact, and the reader can still see that 123 pages and 123 King Ave were never the same reading. Written normalization is the polished line, the one a clinical note ends up with, where the decision has already been made and is no longer visible. Structured extraction lifts fields out of the same audio for something downstream to act on, and validates them against a schema rather than against the audio.

The three answer different questions about the same recording, and each is checked in a different way. The first against the tokens, the second against written convention, the third against a schema that never heard the speaker.

Which of those checks matters depends on what the transcript is for. Captions, clinical notes, call analytics and voice forms each ask post-processing for something different.

Punctuation is the case where the cost is easiest to see, because one mark carries the whole speech act. Tilk and Alumäe measured punctuation restoration in 2016 on the IWSLT2011 English test set, roughly 13K words. Their best model reaches an overall F1 of 64.4 on reference transcripts and 61.4 on ASR output. The question mark falls from 66.7 to 54.0 across the same pair. The earlier DNN baselines restored no question marks at all — precision and recall of 0. Their own summary of the gain: “The overall F1-score improves by 8.9% on reference text and by 10.5% on ASR output when comparing our T-BRNN model to the best baseline (DNN-A).”

So the one mark that records whether an utterance was a question is the worst-scoring class, and it degrades further once the input is real recognizer output rather than a clean reference. A transcript that renders a question as a statement has not misheard a word. It has changed what the speaker did.

All of that argues for one ordering, and the ordering is worth stating as a sequence of moves.

Preserving a raw layer comes first. It is the step that creates the assumptions: the promise that the original tokens, the alternatives and the alignments are still reachable. Applying explicit transformations then spends those assumptions. Every rewrite is permitted only because the thing it replaced was kept.

The industry's own answer to how that second step should be built is unusually blunt. A 2021 paper on NeMo inverse text normalization says it in one line: “Many state-of-the-art ITN systems use hand-written weighted finite-state transducer(WFST) grammars since this task has extremely low tolerance to unrecoverable errors.” Its authors evaluate the open-source NeMo ITN library on a modified version of the Google Text Normalization dataset. The reason for the hand-written grammar is not conservatism about machine learning. A rewrite of this kind is unrecoverable, in the same sense as reading 3 cm as three kilometers. So the stage is built to be inspectable and versioned, rather than merely accurate on average.

Calibrating and exposing confidence is where a team finds out whether the promise survived the workflow.

That last move is also where the evidence comes from. Entity-level normalization accuracy by type reports that the transformations fire correctly, and reports nothing else. It would score a well-formatted line in Zhou et al.'s 7.4% as clean.

The timing evidence is the one most often assumed rather than measured, and it has a published distribution. The Montreal Forced Aligner paper compared forced-aligned boundaries with hand annotation on the Buckeye corpus in 2017. Only 33% of word boundaries fell within 10 ms. 68% fell within 25 ms, with a mean difference of 24.1 ms. The comparison McAuliffe and colleagues draw is the point: “68% of manual/aligned differences are under 25 msec, which is significantly lower than the 90% intertranscriber agreement reported at 26 msec tolerance.” Forced alignment is the same mechanism WhisperX uses to supply word timestamps the recognizer never emitted. And a caption is required to begin when the speech begins.

There is a plain test for whether any of this was done. A transcript format is reversible when another team can walk a formatted line back to the audio and show that the formatting changed what the speaker said: that the digits printed as 123 were spoken as one twenty three, an address and not a page count, and that nothing in the finished line recorded which reading was chosen.

To design one, say what preserving a raw layer assumes, give one counterexample where that assumption fails, and say what calibrating and exposing confidence then does. The counterexample is easy to find. An unrecoverable rewrite — 3 cm read as three kilometers — leaves no trace in the text it replaced. That is exactly why NeMo's authors keep the stage in versioned grammars a person can read.