Qwen-Audio-3.1-ASR Technical Report

AI summary

Overview

• Research area: Speech and audio-language modeling (cs.SD) — automatic speech recognition (ASR), speaker-diarized transcription, and general audio understanding built on a large language model backbone.
• Technical level: Advanced. The report assumes familiarity with encoder–adapter–LLM architectures, CTC auxiliary branches, LoRA, Mixture-of-Experts models, GRPO reinforcement learning, and ASR metrics such as WER, DER and cpWER.
• Scope (one sentence): A technical report describing the Qwen-Audio-3.1-ASR family — 30 languages, 16 Chinese dialects, five deployment variants — which extends its predecessor with speaker-diarized transcription (speaker labels plus timestamps) and, in its Next variant, general audio understanding over speech, environmental sound and music.

What This Paper Is About

A plain transcript captures only part of what an audio recording contains: in multi-party conversations you also need to know who spoke and when, and in many recordings you need to interpret environmental sounds and music that words alone cannot convey. This report describes a family of audio-language models that addresses all three questions — what was said, who said it and when, and what the audio conveys beyond its transcript — while retaining instruction-controlled recognition features such as historical-context conditioning, hierarchical hotwords and transcription polishing. The family spans variants for asynchronous file transcription, low-latency streaming, interactive voice input and non-streaming analysis, with speaker-diarized ASR and audio understanding restricted to the non-streaming variants.

Key Contributions

1. Enhanced core speech recognition. The model builds on the instruction-controlled recognition framework of Qwen-Audio-3.0-ASR, retaining 30 languages and 16 Chinese dialects along with contextual conditioning, two-tier hotword customization, and native transcription polishing, and strengthening recognition across multilingual, dialectal and application-oriented settings.
2. Speaker-diarized ASR. Transcription is extended to structured output: each segment carries a recording-local speaker label, a start–end time interval in seconds, and the recognized text, serialized as [SPK|start-end]text. The output keeps speaker labels consistent across turns and permits overlapping intervals for different speakers.
3. General audio understanding (Next variant only). Qwen-Audio-3.1-ASR-Next is trained on audio captioning (speech, environmental sound, music), temporal grounding (single-interval boundary localization and multi-interval retrieval), and audio question answering in both open-ended and multiple-choice formats.
4. A five-variant deployment family. Flash-Filetrans, Flash-Streaming, Flash-Message, Flash and Next each expose a different capability subset, making recognition quality, output structure, audio understanding and deployment configuration explicitly separable concerns.

Main Findings

• Supervision is imbalanced across token categories in speaker-diarized training. Speaker-ID number tokens are 2.90% of supervised tokens but only 1.97% of the unweighted cross-entropy loss; timestamp tokens are 33.44% of tokens and 49.51% of loss; formatting tokens are 14.48% of tokens and approximately 0.23% of loss; transcription tokens are 46.50% and 47.74%; the remaining 2.68% of tokens and approximately 0.55% of loss are grouped as "other."
• Reweighting speaker-ID tokens helps. Because speaker-ID prediction receives a small share of aggregate supervision despite its role in diarization, the authors multiply the cross-entropy loss on speaker-ID number tokens by 10 while keeping unit weight on timestamp, formatting, transcription and other tokens, without changing target sequences or token counts. They report this substantially improves both DER and cpWER.
• Coarse timestamps train better than fine ones. Initial targets kept original timestamps at 1 ms precision; quantizing them to an 80 ms grid, matching the 12.5 Hz frame rate of the encoder/adapter representations, gives substantially faster convergence of training loss, DER and cpWER, and better DER and cpWER under the same data mixture.
• Recognition improves over the predecessor. The report states improved multilingual recognition relative to Qwen-Audio-3.0-ASR, summarized in Figure 1 as the unweighted mean of language-specific CER/WER across seven languages in GigaSpeechBench.
• Diarization is benchmarked on four sets. Figure 1 reports the unweighted mean cpWER across AISHELL-4, AliMeeting-test, MLC-SLM and MagicData-RAMC.
• Audio understanding is evaluated on MMAU and MMSU. Accuracy on both is measured with the Qwen-Audio-3.1-ASR-Next variant.
• Numeric scores are not in the supplied excerpt. The paper defers full evaluation results to Section 5, which is outside the truncated content provided, so no benchmark values can be quoted here.
• Capability boundaries are deliberate. Speaker-diarized ASR and general audio understanding are unavailable through the Streaming and Message profiles; streaming produces incremental transcripts followed by a non-streaming refresh at the utterance end, and does not support diarization or audio understanding.

Methodology in Plain English

The model reuses an audio-language template: an audio encoder turns the waveform into acoustic representations, an adapter maps them into the language model's input space, and an LLM decoder generates the output token by token, conditioned on the audio plus a text instruction and optional context. A small auxiliary CTC branch off the encoder supplies a preliminary hypothesis used for hotword handling and coarse alignment for long-audio context injection.

Training runs in six stages. First the encoder is pretrained with BEST-RQ self-supervised learning on tens of millions of hours of unlabeled audio, with some Transformer layers initialized from a Qwen3 LLM. Second, the encoder is trained with an attention-based encoder–decoder objective on labeled Chinese and English data, where pseudo-labels come from Paraformer-V2, Whisper-large-v3 and SenseVoice and pass through inverse text normalization; the AED decoder is then thrown away. Third, about 20 million hours of multilingual ASR and interleaved text–audio data train the adapter and LLM jointly with the encoder frozen. Fourth, a cooldown stage uses about one million hours of higher-quality data with rebalanced languages and dialects and a broader task mixture; ASR samples are re-transcribed with VibeVoice and those above 0.3 WER are removed as a quality filter only. Fifth, supervised fine-tuning updates the encoder and adapter while the base LLM is adapted with LoRA, covering specified-language transcription, synthetic and human-recorded domain-entity sentences, polishing data, historical-context data with deliberately noisy or conflicting contexts, two-tier hotwords (P0 high priority, P1 broader pool), and speaker-diarized examples serialized as timestamped segments. Sixth, reinforcement learning with GRPO uses the authors' asynchronous FunVerl framework — audio processor, vLLM rollout engine, task-specific reward module, and Megatron policy trainer connected by a trajectory queue with a policy-staleness threshold — optimizing rewards for accuracy, keyword precision/recall, hotwords, context entities, hallucination suppression, and for diarization DER, cpWER and temporal consistency.

Why This Matters

Recognizing words is only one of three questions practical audio systems must answer, and this report treats output structure and audio understanding as first-class parts of the model rather than as separate downstream pipelines. Jointly generating text, speaker labels and timings avoids error accumulation from stitching together independent diarization and recognition stages, and the token-statistics analysis gives a concrete, reproducible recipe for fixing under-supervised output fields.

Real-world applications implied by the described capabilities:

• Meeting and interview transcription, where the structured output preserves who spoke when and handles overlapping speech and rapid turn-taking.
• Real-time voice interfaces and dictation, served by the streaming and interactive Message variants with incremental transcripts.
• Media and film workflows, since film and television recordings with original subtitles were used to build speaker-diarized training data.
• Domain-specific recognition in finance, healthcare, IT and artificial intelligence, supported by entity-focused augmentation and hierarchical hotwords.

Industry relevance: the five-variant split (file, streaming, message, non-streaming, analysis) maps model capabilities onto concrete serving constraints, and the MoE backbone is chosen to keep per-token computation and decoding cost low enough for latency-sensitive streaming.

Future Directions

• Extending diarization to streaming. The report explicitly excludes speaker diarization and audio understanding from the Streaming and Message profiles; closing that gap is the obvious next step for live multi-party use.
• Refining temporal supervision further. The 80 ms grid was chosen to match the encoder's 12.5 Hz frame rate; whether finer or learned temporal discretizations help once speaker-token reweighting is in place remains open.
• Broadening the audio-understanding task set. Grounding is currently split into boundary localization and multi-interval retrieval, with captioning and QA on MMAU and MMSU; scaling to more compositional reasoning over overlapping events is untested here.
• Quantifying the trade-offs across variants. The report does not, in the supplied content, report per-variant results, so how much recognition quality is sacrificed in streaming versus non-streaming configurations is not established.

Target Audience

Speech engineers and applied researchers building production ASR and audio-understanding systems, especially those who need speaker-attributed, timestamped transcripts or a single model serving multiple latency profiles. It is also useful for teams working on reinforcement learning for sequence-level ASR objectives and for anyone interested in the practical engineering details — data filtering thresholds, loss reweighting, timestamp quantization — that technical reports often omit. Readers seeking benchmark score tables will need Section 5, which is not included in the provided excerpt.

Authors’ abstract

We present Qwen-Audio-3.1-ASR, a family of audio-language models that builds upon its predecesor Qwen-Audio-3.0-ASR with improved speech recognition and expanded capabilities in speaker-diarized transcription and general audio understanding. The family retains support for 30 languages and 16 Chinese dialects, together with instruction-controlled capabilities including historical-context conditioning, hierarchical hotword customization, and native transcription polishing. Speaker-diarized ASR produces structured outputs containing speaker labels, timestamps, and recognized text, maintaining consistent speaker labels across turns and the representation of overlapping speech. The general audio understanding capabilities cover audio-, speech-, and music-captioning, temporal grounding and sound event detection, as well as audio question answering and multiple-choice reasoning. These tasks broaden the application scope from recognizing spoken words to interpreting speakers, acoustic events, and their temporal relations. To accommodate different deployment requirements, the family includes variants for asynchronous file transcription, low-latency streaming, interactive voice input, and non-streaming recognition, with speaker-diarized transcription and general audio understanding available in certain variants. Evaluations span conventional ASR, multilingual and dialectal speech, speaker-diarized transcription, and general audio understanding, demonstrating the capabilities of the Qwen-Audio-3.1-ASR family across transcription and broader audio analysis tasks.