Qwen3-ASR Technical Report

AI summary

Overview

• Research area: Automatic speech recognition (ASR) and speech-text forced alignment, built within the Large Audio-Language Model (LALM) paradigm.
• Technical level: Intermediate. The high-level results are accessible, but the training pipeline, the AuT encoder design and the non-autoregressive alignment formulation assume some familiarity with modern speech and language-model architectures.
• Scope in one sentence: This technical report introduces the Qwen3-ASR family — Qwen3-ASR-1.7B and Qwen3-ASR-0.6B all-in-one recognizers covering 52 languages and dialects, plus Qwen3-ForcedAligner-0.6B, an LLM-based non-autoregressive timestamp predictor covering 11 languages — and evaluates them on public and internal benchmarks against proprietary APIs and open-source baselines.

What This Paper Is About

The paper describes a family of speech models that must do two things at once: transcribe speech into text across many languages and dialects, and predict accurately when each word or character was spoken so that transcripts can be turned into subtitles or captions. Traditional ASR systems treat these as separate problems — recognition in one model, timestamping as a CTC- or CIF-based post-processing step — and they degrade in noisy, accented, dialectal, or long-form audio. The authors' goal is a single model family that recognizes 30 languages and 22 Chinese dialects with language identification built in, plus one lightweight forced aligner that timestamps text-speech pairs in 11 languages without language-specific phoneme dictionaries.

Key Contributions

1. State-of-the-art all-in-one ASR and LID. Qwen3-ASR-1.7B and Qwen3-ASR-0.6B jointly perform language identification and recognition for 30 languages and 22 Chinese dialects, with claimed robustness to noisy environments, singing voice, songs with background music, and complex text patterns.
2. A novel LLM-based forced-alignment architecture. Qwen3-ForcedAligner-0.6B is presented as the first large-language-model-based speech forced aligner producing timestamps at flexible granularities (word, sentence, paragraph). Unlike the Montreal Forced Aligner (MFA) and NeMo Forced Aligner (NFA), it is a single unified multilingual system rather than a set of per-language models.
3. A unified training pipeline and open release. The models are post-trained from Qwen3-Omni through AuT pretraining, Omni pretraining, ASR supervised fine-tuning, and ASR reinforcement learning with Group Sequence Policy Optimization (GSPO).
4. An open-source toolkit. Model weights plus a codebase supporting multi-granularity alignment, streaming transcription, multilingual processing, and a reproducible fine-tuning recipe, all released under the Apache 2.0 license.

Main Findings

• Recognition accuracy. Qwen3-ASR-1.7B is reported as achieving state-of-the-art performance among open-source ASR models and as competitive with the strongest proprietary APIs (GPT-4o-Transcribe, Gemini-2.5-Pro, Doubao-ASR). Scaling from 0.6B to 1.7B produced clear, stable gains.
• Chinese and dialect strength. The models rank at or near the top on Mandarin benchmarks and handle Cantonese and other dialect datasets, including long-utterance settings. On WenetSpeech the paper states Qwen3-ASR outperforms available baselines by a large margin.
• Internal robustness suite. On an in-house suite covering 16 accented-English groups, 22 Chinese dialects, elderly and children's speech, extremely low SNR, tongue-twisters, and multi-speaker Mandarin conversation, Qwen3-ASR-1.7B scored best across all evaluated Mandarin subsets and on both conversational Cantonese and the aggregated 22-dialect evaluation.
• Multilingual limits are acknowledged. Qwen3-ASR-1.7B leads on the 12-language and 20-language Fleurs subsets, but its performance degrades on the full 30-language Fleurs setting relative to Whisper-large-v3, which the authors attribute to long-tail languages.
• Language identification. Average LID accuracy: Qwen3-ASR-1.7B 97.9%, Qwen3-ASR-0.6B 96.8%, versus Whisper-large-v3 at 94.1%. Most remaining Fleurs errors come from confusion between Malay (ms) and Indonesian (id).
• Singing and songs. Qwen3-ASR-1.7B achieved the best result on M4Singer, MIR-1k-vocal, and Popcs, and was second to FunASR-MLT-Nano on Opencpop. On the long-form EntireSongs sets it scored 14.60 WER on English and 13.91 on Chinese, while Whisper-large-v3 and FunASR-MLT-Nano are marked N/A for those sets.
• Streaming degradation is modest. With a 2-second chunk size, a 5-token fallback, and the last four chunks unfixed, Qwen3-ASR-1.7B's average error rose from 2.69 (offline) to 3.33 (streaming); Qwen3-ASR-0.6B went from 3.48 to 4.40.
• Inference efficiency. Qwen3-ASR-0.6B reaches an average TTFT as low as 92ms, RTF as low as 0.064, and throughput as high as 2000 at a concurrency of 128. Qwen3-ForcedAligner-0.6B maintains an RTF close to 0.001 even under high concurrency, and the paper states it can process 1,000 seconds of audio per second.
• Timestamp accuracy. Qwen3-ForcedAligner-0.6B shows a relative reduction of 67%–77% in accumulated average shift on human-labeled test datasets compared with competing forced-alignment methods. Its average AAS is 42.9 ms on MFA-labeled raw data, 52.9 ms on MFA-labeled Concat-300s, and 32.4 ms on human-labeled data. Baselines degrade sharply on long utterances — NFA reaches 246.7 ms and WhisperX 2708.4 ms on MFA-labeled Concat-300s.

Methodology in Plain English

The models are built by post-training from an existing omni-modal foundation model, Qwen3-Omni. Audio first passes through a separate speech encoder called AuT, which is an attention-encoder-decoder ASR model that downsamples 128-dimensional Fbank features by a factor of eight, giving a 12.5Hz token rate. A dynamic attention window between 1 and 8 seconds lets the same model handle both short streaming chunks and long offline queries. The encoder output is projected into the language model, which produces the transcript.

Training proceeds in four stages: AuT pretraining on roughly 40 million hours of pseudo-labeled ASR data (mostly Chinese and English); Omni pretraining on multi-task audio, vision and text data for 3 trillion tokens; supervised fine-tuning on a smaller, disjoint multilingual set that includes non-speech, streaming-enhancement and context-biasing data; and reinforcement learning with GSPO on about 50k utterances (35% Chinese and English, 35% multilingual, 30% functional). A deliberate design choice is that the models do not follow natural-language instructions, to avoid instruction-injection and instruction-following failures — the output is either a language tag plus transcript, or a "None" language tag when no speech is detected.

The forced aligner reformulates alignment as slot filling. The transcript is annotated with [time] special tokens at word or character boundaries, and the model directly predicts a discretized timestamp index for each slot. Timestamps are discretized by dividing by the 80ms frame duration of the AuT output, with a maximum of 3,750 classes to support up to 300 seconds of audio. Training uses causal, non-shifted sequences with cross-entropy computed only in the timestamp slots, plus random dynamic insertion of slots to improve generalization. Because supervision is expensive, the training targets come from MFA pseudo-labels — the paper emphasizes that the model distills and smooths these noisy labels rather than replicating them. At inference, all slots are predicted at once in a non-autoregressive pass, and multiplying indices by 80ms recovers the timestamps.

Evaluation combines public benchmarks with in-house suites, because the authors argue ASR models may differ little on open-sourced benchmark scores while exhibiting large quality differences in real-world scenarios.

Why This Matters

Research impact. The paper argues that LALM-based ASR handles long-form transcription, noise robustness, named-entity recognition, and multilingual coverage more naturally than traditional Transducer or attention-encoder-decoder systems, and that LALM-based timestamping can be both more accurate and faster than CTC/CIF post-processing. If the reported results hold, it moves forced alignment from a per-language tooling problem to a single multilingual model problem, and it explicitly targets the ceiling of annotation errors on existing test sets by introducing internal benchmarks.

Real-world applications:

• Subtitles and captioning — word- and sentence-level timestamps across 11 languages from one model, without per-language alignment models.
• Meeting and broadcast transcription — WenetSpeech-style noisy meeting speech and internal News-Multilingual broadcast speech are explicitly evaluated.
• Singing and music content — karaoke lyric alignment and transcription of complete songs with background music, where Whisper-large-v3 and FunASR-MLT-Nano are marked as not supporting long-form songs.
• On-device and high-throughput serving — Qwen3-ASR-0.6B's 92ms TTFT and throughput of 2,000 seconds of audio per second at concurrency 128 target deployment rather than research-only use.
• Speech dataset labeling — the forced aligner is framed as enabling scalable labeling of speech-transcript pairs.

Industry relevance. The release under Apache 2.0, the claim of parity with proprietary APIs, and the measured vLLM throughput make this directly relevant to teams deciding between commercial ASR APIs and self-hosted models. The model sizes (0.6B and 1.7B) and the sub-1B positioning are aimed squarely at cost-sensitive and edge deployments.

Future Directions

• Long-tail multilingual coverage. The reported regression versus Whisper-large-v3 on the full 30-language Fleurs setting is an explicit open problem the authors flag as room for improvement.
• Mitigating systematic alignment bias. Training on MFA pseudo-labels introduces inherent noise and systematic shifts; the paper claims distillation and smoothing reduce this, but whether the aligner can be pushed further with cleaner or human-verified supervision remains an open question.
• Beyond the current modality and duration limits. The ASR models handle single speech up to 20 minutes and the aligner up to 300 seconds — scaling both while retaining accuracy on the concatenated long-form cases (where Qwen3-ForcedAligner still shows AAS of 52.9 ms and 24.8–42.5 ms on human-labeled long-form sets) is a natural next step.
• Closing the streaming–offline gap. Streaming inference raises average error relative to offline on every reported test set; narrowing that gap without sacrificing the low TTFT is left for future work.
• Extending the family. The conclusion states the team will continue to advance the open model family in both accuracy and functionality.

Target Audience

This paper is most useful to speech and audio-ML engineers, ASR product teams, and researchers working on multilingual recognition or forced alignment. It will particularly benefit readers who need to make build-versus-buy decisions about ASR, since it directly compares open models against GPT-4o-Transcribe, Gemini-2.5-Pro and Doubao-ASR, and readers interested in subtitle or captioning pipelines, for whom the forced-alignment design and its timestamp accuracy numbers are the central contribution. Because it is a technical report rather than a novel-architecture paper, it assumes familiarity with standard ASR pipeline concepts but does not require deep expertise in the underlying training mathematics.

Authors’ abstract

In this report, we introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model. Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects. Both of them leverage large-scale speech training data and the strong audio understanding ability of their foundation model Qwen3-Omni. We conduct comprehensive internal evaluation besides the open-sourced benchmarks as ASR models might differ little on open-sourced benchmark scores but exhibit significant quality differences in real-world scenarios. The experiments reveal that the 1.7B version achieves SOTA performance among open-sourced ASR models and is competitive with the strongest proprietary APIs while the 0.6B version offers the best accuracy-efficiency trade-off. Qwen3-ASR-0.6B can achieve an average TTFT as low as 92ms and transcribe 2000 seconds speech in 1 second at a concurrency of 128. Qwen3-ForcedAligner-0.6B is an LLM based NAR timestamp predictor that is able to align text-speech pairs in 11 languages. Timestamp accuracy experiments show that the proposed model outperforms the three strongest force alignment models and takes more advantages in efficiency and versatility. To further accelerate the community research of ASR and audio understanding, we release these models under the Apache 2.0 license.