VibeVoice-ASR-Streaming Technical Report

AI summary

Overview

• Research area: Automatic speech recognition (ASR) — specifically end-to-end streaming speaker-attributed ASR built on a large language model backbone (eess.AS).
• Technical level: Advanced. The paper assumes familiarity with autoregressive LLM decoding, speech tokenizers, diarization metrics (cpWER/cpCER), and streaming latency accounting.
• Scope in one sentence: This technical report introduces VibeVoice-ASR-Streaming, an LLM-based end-to-end system that emits "who said what" incrementally as audio arrives, without a separate diarization stage, and benchmarks it against deployed streaming ASR services and its own non-streaming predecessor.

What This Paper Is About

End-to-end models such as VibeVoice-ASR unified transcription and speaker diarization in a single model, but they read the entire recording before emitting output, which is incompatible with the low-latency demands of real-time voice assistants and agents. The core problem is that speaker attribution places a harder requirement on retained context than ordinary ASR does: a speaker who first appeared minutes earlier must still receive the same label, so history is not merely helpful but is what fixes speaker identities across a conversation. The goal is a single model that interleaves fixed-size audio chunks with generated speaker-attributed text so that it produces a labeled transcript as the conversation unfolds, rather than adding a separate streaming diarizer or speaker cache.

Key Contributions

1. One of the first investigations of end-to-end LLM-based streaming speaker-attributed ASR, showing that interleaved speech-text generation can support long-form streaming recognition with strong recognition and speaker-attribution performance. The 1.5B and 7B model weights and inference code are released (github.com/microsoft/VibeVoice, with a demo at microsoft/VibeVoice-ASR-Streaming and checkpoints at microsoft/VibeVoice-Collection).
2. A study of the key design choices for LLM-based speaker-attributed streaming ASR, covering chunk size (15 vs. 22 latent frames), lookahead depth (0, 2, 4 frames), model scale (1.5B vs. 7B), and speaker-label placement (label before vs. after segment text).
3. Detailed comparisons against the non-streaming VibeVoice-ASR checkpoint it is initialized from and against deployed streaming systems (Gemini 3.5 Transcribe Live, GPT Realtime Whisper, GPT Live Transcribe, ElevenLabs Scribe v2 Realtime, Azure ConversationTranscriber, and Google Cloud Speech-to-Text).
4. Serving-cost analysis over long recordings, including real-time factor measurements on up to 480 seconds of audio.

Main Findings

• Recognition-only performance: The 7B 22-frame configuration achieves the lowest five-set mean (24.66) among the compared streaming systems, against 25.23 for Gemini 3.5 Transcribe Live, 39.31 for GPT Realtime Whisper, 40.55 for GPT Live Transcribe, and 41.39 for ElevenLabs Scribe v2 Realtime. It is best on AISHELL-4 (22.76 vs. Gemini's 30.09), AliMeeting (33.83 vs. 35.29), and AMI-IHM (19.83 vs. 19.85), while Gemini 3.5 Transcribe Live is best on AMI-SDM (27.18 vs. VibeVoice's 29.81). On the nine-language MLC-Challenge macro average, Gemini leads at 13.73 and VibeVoice is second at 17.09.
• Speaker-attributed performance: Across the 13 speaker-attributed settings in Table 2, the model gives the best or tied-best cpWER/cpCER on 12. It improves over Azure ConversationTranscriber by 2.39 to 12.45 points on the four meeting benchmarks and moves the MLC-Challenge average from 27.06 to 22.75.
• Latency advantage: VibeVoice-ASR-Streaming commits to a speaker label after an expected 2.00 s (the algorithmic delay C/2 + T_lookahead at 22 frames). Azure ConversationTranscriber measures 8.21 s and Google STT 9.12 s when using the label at first emission, and 51.06 s when using the final revised label after the full recording is processed.
• Google STT revises labels heavily: Its labels keep changing for a further 16.5–31.3 s after text settles, 42.6–67.9% of words have their label revised at least once after first emission, and revising labels reduces cpWER by 27.0 to 32.2 points relative to the first-emitted label.
• Streaming conversion costs more in speaker attribution than in transcription: Against the non-streaming VibeVoice-ASR checkpoint, WER/CER rises by 0.75 to 3.53 points while cpWER/cpCER rises by 5.13 to 6.67 points on every benchmark.
• Larger chunks help, and help speaker attribution more: Going from 15 to 22 latent frames improves the five-set mean by 1.46 WER/CER and 4.06 cpWER/cpCER at 7B, and by 1.31 and 2.91 at 1.5B. Moving from 1.5B to 7B at a fixed chunk size lowers the mean cpWER/cpCER by 12.76 points at 22 frames and 11.61 at 15 frames, versus 4.69 and 4.54 for WER/CER.
• Speaker-label placement is essentially neutral: Emitting the label before the segment text (head, the released configuration) versus after (tail) lands on the same mean cpWER/cpCER, 31.55 against 31.56, with the tail trailing by 0.98 on WER/CER. The paper reads the two placements as equivalent in accuracy within the measurement margin, and notes that a tail label buys nothing unlike what a cascaded embed-and-cluster system would predict.
• Lookahead monotonically helps: At a fixed 22-frame chunk, both metrics improve strictly with L on all five benchmarks. The first pair of frames (L=0 to L=2) is worth 1.27 WER/CER and 1.80 cpWER/cpCER on average, the second pair (L=2 to L=4) is worth 1.24 and 2.54, so speaker attribution is still accelerating at L=4. Each step costs 0.267 s of expected latency; the L=4 configuration gives 2.00 s versus 1.47 s at L=0.
• Single-speaker short-form audio is not the target use case: The model wins no individual test set in Table 3 but stays close on every one. With 22-frame chunks it is second on AISHELL-1 (4.01), LibriSpeech test-clean (2.33), and test-other (6.49), with a four-set mean of 5.76 against X-ASR's 5.77 and Voxtral-Mini-4B-Realtime's 6.41. The 22-frame configuration beats the 15-frame one on all four sets.
• Real-time factor is comfortably below 1: On a single A100 80GB PCIe under vLLM in bfloat16 at batch size one, the 7B 15-frame configuration decodes a chunk in 146 to 208 ms against a 2000 ms chunk, giving an RTF at or below 0.104. At 480 s of audio, decoding 240 chunks takes 49.94 s (RTF 0.104).

Methodology in Plain English

The system is built on the existing VibeVoice-ASR model. Speech is turned into two parallel streams of compact representations by pre-trained tokenizer encoders — one acoustic, capturing spectral detail, and one semantic, aligned with linguistic content. At 24 kHz, these produce one latent frame every 133.3 ms, so chunk sizes are expressed in frames and every latency figure is a multiple of 133.3 ms.

The key idea is to write the audio and the transcript as one interleaved sequence: a speech chunk, then its speaker-attributed text, then the next speech chunk, and so on. Because previously seen audio and generated text stay in the model's context, each new chunk is decoded against the accumulated conversation history, which is what keeps a speaker's label consistent over time. Before generating a chunk's text, the model is also allowed to read a small fixed lookahead of 4 latent frames (about 0.5 s) so it can resolve boundaries. Text generation ends when the model emits a special end-of-chunk token, which hands control back to the audio stream. Speaker labels are ordinal, assigned in order of first appearance, and written inline in plain text (for example, "\n Speaker 0:") with no timestamps. Optional context such as names, technical terms, abbreviations, and hotwords can be supplied before decoding and stays available throughout the session.

Training data preparation uses Qwen3-ForcedAligner-0.6B to obtain word-level timing, after which the reference transcript is split into per-chunk targets. Part of the mixture is synthetic: meeting-style multi-speaker conversations with domain terminology and proper nouns, where the spoken form (used for synthesis) and written form (used as the ASR target) are kept separate so numbers and abbreviations are spoken naturally but transcribed canonically. The synthetic speech is then augmented at the waveform level by overlapping speakers and convolving with room impulse responses, yielding 50,884 recordings totaling 4,519.6 hours.

Training proceeds in three stages that differ only in sample construction, not in model or objective. Stage 1 trains offline on full recordings to establish multi-speaker recognition and attribution. Stage 2 switches the sample construction to the interleaved format, drawing on roughly 420,000 hours of English and Chinese speech. Stage 3 fine-tunes on a curated mixture of about 13,000 hours drawn from public training splits and the synthetic multi-speaker data, settling transcription conventions, consistent speaker labeling, and hotword following. Each streaming configuration is initialized from the non-streaming checkpoint of the same scale, and the 15- and 22-frame configurations are trained independently. Stage 3 uses AdamW (β1=0.9, β2=0.95, weight decay 0.1), gradient clipping at 2.0, bfloat16, a cosine schedule with peak learning rate 5×10⁻⁵, sequences packed to 8,192 tokens, and is a 500-step run on eight GPUs at a global batch of 64 sequences with 35 warmup steps; released checkpoints are taken at step 400.

Evaluation follows the MeetEval protocol, reporting WER (recognition alone) and cpWER (concatenated minimum-permutation WER, which incorporates speaker assignment). Chinese, Japanese, and Korean are scored at the character level as CER and cpCER. Datasets are AISHELL-4, AliMeeting, AMI-IHM and AMI-SDM, and nine MLC-Challenge languages (English, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish), with evaluation recordings capped at 480 seconds. Single-speaker checks use AISHELL-1, LibriSpeech test-clean and test-other, and GigaSpeech. The benchmark MLC-Challenge covers more languages than these nine, but the forced aligner does not support them, so the remaining languages are absent

Authors’ abstract

Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.