Towards Efficient Simultaneous Inverse Text Normalization with Pretrained Text-to-Text Language Model and Read-Tag-Write Policy

AI summary

Overview

• Research area: Natural Language Processing, specifically streaming (simultaneous) Inverse Text Normalization for Automatic Speech Recognition output.
• Technical level: Advanced. The paper assumes familiarity with encoder–decoder Transformers, attention masking, simultaneous machine translation read/write policies, and key-value caching.
• Scope: The paper proposes and evaluates the first streaming end-to-end ITN system, built by adapting the pretrained text-to-text model EnViT5 with a Read–Tag–Write decoding policy, and benchmarks it on Vietnamese datasets against non-streaming end-to-end, hybrid streaming, and wait-k baselines.

What This Paper Is About

Inverse Text Normalization (ITN) turns the spoken-form text emitted by ASR systems (for example, "one point two dollars") into readable written form (for example, "$1.20") by restoring punctuation and capitalization and converting semiotic and phonetic phrases. Existing streaming ITN systems are hybrid, pairing neural taggers with expert-written finite-state transducer (FST) rules, which limits scalability across domains and languages, while standard end-to-end seq2seq models cannot stream because their encoder–decoder architectures rely on global cross-attention. The paper's goal is an end-to-end ITN model that processes text strictly left to right, matches non-streaming accuracy, beats hybrid streaming methods, and meets real-time latency requirements without any manual FST rules or IOB labeling.

Key Contributions

1. A recipe for making pretrained seq2seq models streamable. The authors combine architectural changes (Dynamic Right Context masking in the encoder), a streaming-aware multi-task objective over a tagging loss and a generation loss, and a new on-the-fly data augmentation called Prefix-based Training Augmentation, so that a pretrained model can perform ITN with strict left-to-right processing.
2. An efficient Read–Tag–Write (RTW) decoding policy. The policy reads incoming ASR chunks, tags tokens with a 3-class {B, I, O} tagger, passes O-tagged (verbatim) tokens straight through, and triggers the autoregressive decoder only when a normalization span completes. It is accelerated by encoder and decoder KV caching plus a Copy–Paste strategy that retains historical context while removing redundant computation.
3. A newly curated Vietnamese ITN dataset. The dataset is built from the binhvq news corpus (approximately 6 billion words) and contains 5M training sentence pairs, 50k validation pairs, and 50k test pairs, with deterministic phrase-level spoken–written alignments, token-level normalization boundary annotations, IOB tags, and a phonetic dictionary of approximately 125k common foreign and OOV terms.
4. Comprehensive empirical evaluation. The work reports I-WER, NI-WER, WER, tagging F1, bootstrap 95% confidence intervals over 10,000 iterations, and per-chunk latency benchmarking against non-streaming end-to-end, hybrid streaming, and wait-k baselines, including a real-world evaluation on noisy ASR transcripts.

Main Findings

• Pretraining matters for both offline and streaming ITN. Using pretrained EnViT5 weights (B1) instead of training from scratch (B0) gave relative WER improvements of 16.70% [10.45, 26.15] With PnC and 33.33% [5.33, 63.16] Without PnC. In streaming, the pretrained S1 beat the from-scratch S0 by 18.94% [10.92, 27.84] and 52.60% [29.30, 67.08] in the same two settings.
• The auxiliary tagging loss produces the strongest non-streaming baseline. B2, which adds the tagging loss to B1, reached an F1 of 97.34 and WERs of 3.63% (With PnC) and 0.52% (Without PnC).
• The proposed streaming model is close to non-streaming quality. The S2 variant achieved a tagging F1 of 97.19, I-WER of 12.91%, NI-WER of 1.36%, and WER of 4.18% With PnC, and I-WER 6.24%, NI-WER 0.19%, WER 0.64% Without PnC. The causal-masked S_C with non-streaming inference performed comparably to B2, with marginal WER increases of only 0.16% [0.03, 0.27] and 0.08% [-0.03, 0.18].
• Streaming costs are modest. S2 incurred a relative WER increase of 8.85% [-4.12, 16.77] over B1 With PnC and improved by 13.51% [-38.67, 50.62] Without PnC; relative to B2 the increases were 15.15% [10.29, 17.17] and 23.07% [0.65, 47.15].
• It outperforms hybrid streaming and wait-k baselines. S2 beat the hybrid H2 [12] by 1.98× [1.94, 2.01] and 6.56× [5.74, 7.40] in WER, and the wait-k model [15] (decoded with k = 9) by 1.67× [1.60, 1.75] and 5.81× [4.96, 6.73]. H2's I-WERs were 28.39% and 47.36%, which the authors attribute to the limits of manually crafted FST rules.
• The AdapITN baseline generalizes poorly. AdapITN [2], which uses encoder-only pretrained EnViBert and decodes isolated spans without surrounding context, produced an I-WER of 156.88% and WER of 40.08% With PnC; even Without PnC its 1.45% WER was 2.79× [2.61, 3.30] higher than the B2 baseline.
• Look-ahead context consistently helps. S2 reduced relative WER by 19.31% [18.62, 19.89] and 21.95% [17.95, 25.04] over the zero-look-ahead S1; S4 reduced it by 18.88% [18.50, 19.29] and 22.47% [20.62, 23.99] over S3; the hybrid H2 beat H1 by 18.60% [18.33, 18.93] and 9.09% [8.79, 9.42].
• Inference optimizations deliver a 3.43× speedup. With batch size 100 and sequences of 100–200 words on an NVIDIA H100 GPU (fixed look-ahead r = 1), encoder KV caching cut encoder latency from 54 ms to 10 ms, and combining all optimizations brought total latency at chunk size c = 5 from 206 ms (baseline) to 60 ms. Decoder caching improved total per-chunk latency by 31.82% relative to encoder caching at c = 5, and the 95th percentile latency was 165.5 ms for the fully streaming configuration versus 249.0 ms for encoder caching only. Since a chunk of 5 tokens typically represents 500 ms of audio, the authors state this supports real-time processing and 100 concurrent users.
• Fully streaming decoding costs almost nothing in accuracy. S4, which adds decoder KV caching, differed from S2 by absolute WER increases of only 0.03% [-0.05, 0.09] and 0.05% [-0.03, 0.11].
• Real-world noisy ASR evaluation. On 50k test samples synthesized with Microsoft Edge TTS and transcribed by the ChunkFormer ASR model at 14.21% WER, S4 achieved I-WER 37.08%, NI-WER 2.24%, and WER 10.74%; S2 achieved 37.55%, 2.29%, and 10.89%. Non-streaming B1 was the strongest at 34.09%, 2.24%, and 10.01%. H2 (47.56%, 2.75%, 13.67%), wait-k (41.16%, 2.48%, 11.91%), and S0 (46.83%, 2.51%, 13.30%) were weaker.
• Truncation caveat. The supplied paper content ends mid-sentence in the discussion of Table 2, so the authors' full interpretation of the real-world results and any conclusions after that point are not reported in the available text.

Methodology in Plain English

The authors start from EnViT5, a bilingual Vietnamese–English text-to-text model, and adapt it in four ways.

First, they make the encoder streamable. The encoder is trained with Dynamic Right Context masking, which limits how much future text each token can see. During training the allowed chunk sizes vary from 1 to 5 tokens and the look-ahead varies from 0 to 2 tokens, so the model is exposed to many different latency budgets and can be run at any of them later.

Second, they attach a small linear tagging head on top of the encoder that labels each token as Beginning, Inside, or Outside a normalization span. This uses only 3 classes, unlike hybrid systems that need fine-grained category tags to select FST rules. The whole model is trained with a combined loss: a cross-entropy tagging loss plus the usual sequence generation loss.

Third, they define the Read–Tag–Write inference policy. Incoming ASR text arrives in chunks. The encoder processes each chunk using cached history plus a limited look-ahead and appends the new hidden states to a running sequence. The tagger labels the new tokens. Tokens tagged Outside are passed straight to the output without touching the decoder, minimizing latency and preventing the decoder from altering text that should not change. Only when a B–I span is complete (the next tag is B or O, or a sentence boundary appears) is the decoder triggered, and consecutive completed spans in a chunk are batched into one decoder call. Cross-attention is restricted to the encoder states up to the end of the last completed span, so future text cannot influence the current transformation.

Fourth, they solve a training–inference mismatch. During training, the end-of-sequence token normally depends on a global source boundary, but streaming inference must terminate local spans mid-sentence. Their Prefix-based Training Augmentation (Algorithm 1) randomly truncates training examples with probability p_trunc = 0.8, omitting the source-side <EOS> while terminating the aligned target prefix with <EOS>, so the model learns to stop generation from partial encoder states alone. A forced hard-stop of L_max_local = 64 tokens prevents infinite decoding.

For speed, they cache encoder keys and values, cache decoder self- and cross-attention states, and use a Copy–Paste strategy in which skipped verbatim tokens are still fed into the decoder's context so historical information is preserved without recomputation.

The dataset is built with a regex-driven pipeline over the binhvq news corpus. Capitalization and punctuation examples are made by stripping punctuation and lowercasing; semiotic expressions (numbers, dates, times, fractions, currencies, digit sequences, electronic addresses) are detected in written form and converted to spoken-form candidates by rules, with one candidate randomly sampled per example to reflect natural variability; phonetic phrases are written-form tokens that are neither semiotic nor in the ASR vocabulary, mapped back to spoken form using a manually verified grapheme-to-phoneme dictionary of about 125k entries. Four punctuation marks are covered: comma, period, question mark, and exclamation mark.

Training used up to 20 epochs on 4 NVIDIA H100 (80GB) GPUs via Hugging Face Transformers, with AdamW (peak learning rate 1e-4, one-epoch linear warmup, cosine schedule, β1 = 0.9, β2 = 0.999, weight decay 0.01) and a global batch size of 512. Hyperparameters were tuned on a 500k-sample subset of the training set. Baselines included the hybrid architecture from [12] retrained on the same dataset with the authors' own weighted FST module, a multi-path wait-k model [15], and AdapITN [2] trained with the biasing mechanism excluded.

Why This Matters

Research impact. The paper argues it is the first streaming end-to-end ITN model, which shifts streaming ITN away from hybrid tagger-plus-FST pipelines toward data-driven seq2seq systems. It also shows that techniques from Simultaneous Machine Translation do not transfer directly: because ITN is local and order-preserving rather than reordering-heavy, fixed-lag and generative scheduling of the SiMT kind introduce redundant computation, motivating the span-triggered RTW policy instead. The finding that a causal encoder with non-streaming inference costs almost nothing (0.16% and 0.08% WER) also isolates how much accuracy is lost to masking versus to streaming decoding itself. Because the method relies only on standard text-to-text foundations with no language-specific heuristics, the authors claim it generalizes across languages and is compatible with any seq2seq architecture.

Real-world applications.

• Live ASR captioning and meeting transcription, where formatted text must appear as speech happens to preserve conversational flow.
• Speech translation pipelines, where readable source or target text feeds downstream machine translation and natural language understanding.
• Voice assistants and telephony/dictation systems that must display or act on numbers, dates, currencies, and addresses in written form.
• Large-scale deployments needing many concurrent streams, which the reported 95th-percentile latency of 165.5 ms and claim of serving 100 concurrent users is designed to address.

Industry relevance. The work removes two major production costs of hybrid streaming ITN: writing and maintaining expert FST grammars, and producing extensive IOB annotations per ITN category. The reported 3.43× speedup at chunk size 5 (206 ms to 60 ms) and the real-world noisy-ASR evaluation using Edge TTS and ChunkFormer directly target deployment conditions rather than clean text.

Future Directions

• Broader language coverage. All experiments are on Vietnamese; the authors assert cross-linguistic generalization from standard text-to-text foundations but do not demonstrate it, so testing other languages and scripts is an open step.
• Closing the gap to non-streaming. S2 remained 15.15% [10.29, 17.17] and 23.07% [0.65, 47.15] worse in relative WER than the non-streaming B2, so the remaining loss from unavailable future context is unresolved.
• Resolving the training–inference mismatch in decoder KV caching. The fully streaming configuration introduces a mismatch because training uses the full target prefix while streaming updates states incrementally without re-attending to past tokens under new encoder states; the paper reports only a marginal fluctuation and leaves the mismatch itself unaddressed.
• Adapting SiMT scheduling more systematically. The authors state that the applicability of complex SiMT scheduling to the localized domain of ITN remains unexplored, which suggests further study of adaptive or learned read–write policies for span-structured tasks.
• Understanding the real-world degradation. I-WER rose sharply on noisy ASR input (for example, S4 from 13.22% to 37.08%), and the provided content does not include the authors' full analysis of this drop.

Target Audience

This paper is most useful to researchers and engineers working on speech recognition post-processing, streaming or simultaneous sequence generation, and production ASR systems that need real-time formatted output. It will also interest practitioners evaluating whether to replace hybrid tagger-plus-FST ITN pipelines with end-to-end models, and researchers studying how to make pretrained seq2seq models operate under strict left-to-right constraints. Readers need a working knowledge of Transformer encoder–decoder architectures, attention masking, and inference caching to follow the methodology and latency analysis in detail.

Authors’ abstract

Streaming Inverse Text Normalization (ITN) is vital for converting spoken-form outputs from streaming Automatic Speech Recognition into formatted written text. Existing streaming ITN methods rely on hybrid systems combining neural tagging with expert-crafted finite-state transducer rules, limiting scalability across domains and languages. While end-to-end models offer superior scalability by learning directly from data, standard encoder-decoder architectures are inherently non-streaming due to global attention. In this paper, we propose an efficient streaming end-to-end ITN system, adapting a pretrained text-to-text model to leverage its robust linguistic knowledge. To enable streaming, we introduce architectural adaptations, a specialized training strategy, a Read-Tag-Write decoding policy, and inference optimizations. Experiments on Vietnamese datasets show accuracy comparable to non-streaming baselines, outperforming hybrid methods while satisfying real-time latency requirements.