Confucius4-R2T2: A Stable-Prefix Streaming ASR Model for Real Real-Time Transcription

AI summary

Overview

• Research area: Streaming automatic speech recognition (ASR), specifically converting a large pre-trained full-context speech LLM into a low-latency streaming recognizer whose emitted text is never revised.
• Technical level: Advanced. The paper mixes formal definitions and proofs with large-scale model training and inference-system engineering.
• Scope: The paper introduces the Longest Stable Prefix (LSP) training target and uses it to post-train Qwen3-ASR-1.7B into Confucius4-R2T2, a stable-prefix streaming ASR model with a frozen audio encoder, configurable 80 ms–2 s chunks, and open-sourced weights and code.

What This Paper Is About

Streaming ASR systems emit partial transcripts while audio is still arriving, and they often revise tokens they already emitted once more audio shows up (called flicker, revision, or hypothesis instability). When a downstream consumer — a translator, a caption display, or a voice agent that has already called a tool — acts on a token, a later revision means wasted computation or a wrong action. The paper's goal is a model that never modifies its earlier output while still approaching the accuracy of full-context recognition and keeping emission latency low.

Key Contributions

1. The Longest Stable Prefix (LSP) concept, with proofs. The paper defines the LSP at time t as the longest earlier model output that is entirely a prefix of the full-audio output, and proves it is monotone under the prefix order, consistent with the full-audio output, and maximal among labels that equal a whole earlier output. The authors contrast this with the "all-future" label of McGraw and Gruenstein (2012) and the "final-anchored" label of Hoang et al. (2026), which in general name strings the model never actually produced.
2. LSP learning, a two-stage recipe that streamizes a model without architectural change. Supervised fine-tuning on LSP targets plus a Wait token, followed by GRPO reinforcement learning, converts a full-context model into a stable-prefix streaming model without causalizing the encoder and without attention masks; the audio encoder and projector stay frozen.
3. A streamability metric for choosing a backbone or teacher. The paper defines streamability as the area under the curve (AUC) of normalized LSP length against normalized audio time, and uses it to quantify how much streaming potential a full-context model has before any training.
4. Inference optimizations and open release. Four techniques — encoder/decoder cache reuse, Wait-terminated decoding, bounded context, and pre-fill verification of a revisable suffix — keep per-update cost low, and the authors release weights at huggingface.co/netease-youdao/Confucius4-R2T2 and code at github.com/netease-youdao/Confucius4-R2T2.

Main Findings

• Stability is guaranteed by construction, not patched at inference. The paper defines a stable-prefix streaming model behaviorally: for every audio signal and all t ≤ t′, the released text at t must be a prefix of the released text at t′. Because LSP labels are always whole earlier outputs, learning them makes earlier output final rather than repaired after the fact.
• The base model was chosen by measurement. On 1,860 English utterances from a selected Buckeye Speech Corpus subset, decoded at 80 ms intervals, Qwen3-ASR-1.7B achieved the highest normalized streamability, 0.493, among the compared models (Qwen3-ASR-0.6B, Whisper large-v2, and WhisperRT).
• Causalizing an encoder can hurt stability. WhisperRT, built by fine-tuning Whisper large-v2's encoder with a causal attention mask, had the worst streamability of the compared models — worse than its own full-context base model — because its unstable output produces much shorter LSPs.
• Freezing the encoder was preferred over causalization. The authors cite Krichli et al. (2026) reporting that LoRA-causalizing Whisper's encoder is worse than leaving it intact at chunk sizes above roughly 300 ms, and handle the train–test mismatch in the decoder by learning when to emit Wait instead.
• Streaming cost is tied to new text, not to a fixed per-update budget. At the 160 ms chunk used in the main results the model performs 6.25 updates per second; with Wait-terminated decoding an update that predicts Wait costs a single pre-fill with no serial decoding steps. Qwen3-ASR-1.7B streaming, by contrast, regenerates five tokens at every update, which the paper puts at 31.25 serial steps per second.
• Context is bounded explicitly. Retaining W = 1 completed window plus the active one limits context to 8(W+1) s of audio and its transcript, and each eviction rebuilds the retained window's KV cache once per 8 s.
• Pre-fill verification matches regeneration under greedy decoding. Treating the previous U-token suffix as a draft and checking it during the pre-fill produces the same output as regeneration up to numerical precision, and a suffix that is still correct costs no decoding step.
• The LSP label is deliberately conservative. One diverging token zeroes credit for that step, so the LSP label is never longer than the final-anchored label and biases the model toward later emission; the authors accept this to keep model admissibility and state that they measure the effect in Section 5.1.
• Streaming quality approaches full-context quality, with gains beyond the post-training languages. This is the claim stated in the abstract (Chinese and English post-training, 1,500 hours of audio). The detailed benchmark tables and the numbers behind these claims are not present in the supplied content, which is truncated at the start of Section 8 ("8.1 Exper…"), so specific WER, latency, and benchmark-name results are not reported here.

Methodology in Plain English

The team started from an existing full-context recognizer, Qwen3-ASR-1.7B, which pairs a 300M-parameter AuT audio encoder (hidden size 1,024, 8× downsampling, 12.5 Hz token rate) with the Qwen3-1.7B language model through a projector. Instead of cutting the encoder down to a causal, left-to-right version, they left the encoder and projector frozen and changed only what the decoder is trained to say.

To build training targets, they fed each utterance to the untouched model in 80 ms increments and recorded every partial hypothesis. They then compared each partial hypothesis against what the same model outputs when it hears the whole utterance. A partial hypothesis counts only if all of it agrees with that final output; the longest such hypothesis is the LSP. Runs of consecutive audio endpoints where the output equaled the LSP and kept that prefix unchanged were merged into stable intervals. At training time they cut each sample at a random point: cuts inside a stable interval are labelled with the stable prefix, and cuts in the gaps are labelled with the prefix plus a special Wait token that tells the model to defer.

Training has two stages, both with the encoder and projector frozen. Supervised fine-tuning used one epoch, fused AdamW, a learning rate of 10⁻⁵, 2% warmup with linear decay, and an effective batch size of 64; 10% of examples were complete utterances and 90% were prefix cuts, half of those inside stable intervals and half in gaps, with the encoder attention window sampled between 1 and 8 seconds. Reinforcement learning with GRPO followed, sampling multiple streaming trajectories over one, two, or four turns, scored by a weighted eight-component reward (CER as the primary signal, plus penalties for redundant predictions, repetitions, language inconsistency, and premature continuation) passed through a tanh(3·R_raw) transform.

For serving, the model decodes in chunks of Δ = kδ rather than every 80 ms. Reusing KV caches for completed 8-second encoder windows, ending an update immediately when the first prediction is Wait, evicting old windows together with the text committed before them, and verifying a small provisional suffix during the pre-fill keep each update fast enough for a real-time factor below 1. Training data was augmented with pure-noise replacement, additive noise at 10–35 dB SNR, reverberation from real or simulated room impulse responses, and language-tag flipping, to target hallucination on non-speech, noise robustness, and accidental translation.

Why This Matters

The paper reframes output stability as a behavioral constraint on the whole system — model plus decoding rule — that is orthogonal to accuracy and latency, rather than as a property of a causal encoder or a post-hoc commit rule. That reframing makes stability something a model can be trained for, and it gives the field a measurement tool (streamability) for deciding whether a given pre-trained model is worth streamizing at all. It also shows a practical route to streaming that avoids the accuracy cost of causalization and avoids retraining an audio encoder.

Real-world applications:

• Live captioning, where text on screen should not visibly change while a reader is following it.
• Speech translation, where an emitted token immediately triggers downstream translation work that would have to be redone after a revision.
• Voice AI agents and tool use, where an emitted token can be an irreversible action such as a tool call or a decision to respond.
• LLM-consumed transcription, such as meeting assistants or retrieval pipelines that process each partial token as it appears.

Industry relevance: the model is released openly, requires no architectural change to the base model, and supports a single set of weights for both streaming (with configurable 80 ms–2 s chunks) and full-context operation. Freezing the encoder and projector also cuts trainable parameters and avoids storing their gradients and optimizer states, which lowers the cost of reproducing the adaptation.

Future Directions

• Whether streamability generalizes beyond ASR. The authors state it is an open question whether the metric is a solid gauge for other streamization settings such as simultaneous translation.
• Whether a stronger teacher helps. R2T2 uses self-distillation to avoid teacher–student distribution mismatch; whether a more capable external teacher would produce better streaming models is left open.
• Reducing the conservativeness of the all-or-nothing LSP credit, which biases the model toward later emission, without giving up the guarantee that labels equal real earlier outputs.
• Cutting the remaining recomputation. The bidirectional encoder forces the active window and retained text to be re-prefilled at every update; the paper compares this against a causal interleaved design it did not implement, and the paper also notes that a word-aligned analysis would be needed to confirm that words are predicted before their acoustic ends.

Target Audience

This paper is most useful to ASR and speech-LLM researchers working on streaming and incremental decoding, to engineers building real-time speech products (captions, translation, voice agents) who need non-retracting output, and to practitioners adapting large pre-trained speech models to latency-constrained deployment. It assumes familiarity with encoder–decoder ASR, token-level decoding, WER/CER, and reinforcement learning for language models; readers without that background can still follow the problem framing and the stable-prefix definition.

Authors’ abstract

Streaming speech recognition is increasingly consumed by machines, such as translators and voice agents, that act on each token as it appears. A token that is later revised costs wasted work or a wrong action. In this work, we present Confucius4-R2T2 (short for Real Real-Time Transcription), a stable-prefix streaming ASR model that never modifies its earlier output while maintaining high accuracy and low latency. R2T2 is obtained by post-training Qwen3-ASR-1.7B on 1,500 hours of Chinese and English audio while freezing its pre-trained audio encoder and projector. The adaptation is based on the concept of Longest Stable Prefix (LSP), a new notion that we introduce in this work. We prove that LSP is monotone under the prefix order, consistent with the full-audio output, and maximal among labels that equal a whole earlier output. Learning from LSP labels converts a full-context model into a stable-prefix streaming one without any architectural change. Extensive evaluations show that R2T2 brings streaming recognition close to the base model's full-context quality while delivering stable text with low emission lag. Its gains extend beyond the post-training languages, demonstrating a practical path from full-context ASR to stable streaming. We open-source the model weights and code at https://huggingface.co/netease-youdao/Confucius4-R2T2 and https://github.com/netease-youdao/Confucius4-R2T2.