The Pulse
David AI Finds a Sevenfold ASR Accuracy Gap Across Languages
David AI released DAI-ASR-I18N, a benchmark of 14 speech systems tested on conversational audio in 21 languages. Its results show much wider differences among systems in five Indic languages than in English, and sharply higher diarization e
AI.info Team · Written with AI ·
Speech-recognition systems vary by an average of 25.4 percentage points across five Indic languages in David AI’s new benchmark—nearly seven times the 3.7-point spread it measured across English, Korean, Mandarin, Spanish and Russian. The company published DAI-ASR-I18N on September 30, offering results for 14 systems tested on unscripted conversation in 21 languages.
The finding challenges the comfort of English-only comparisons: systems that perform similarly on English can separate sharply in other languages. David AI’s evaluation also finds that overlapping speech drives errors in both transcription and speaker identification, a persistent problem for recordings where people interrupt or talk at once.
What the 147-hour benchmark measures
David AI built the benchmark from 147 hours of conversations between pairs of native speakers. Participants discussed open-ended topics in sessions lasting roughly 15 minutes. Each speaker was recorded on a separate, synchronized channel, allowing the same audio to support both automatic speech recognition (ASR) and diarization—the task of identifying who spoke when.
Unlike transcripts that omit disfluencies, the benchmark keeps fillers, repetitions and false starts. David AI says each transcript was reviewed by two human annotators and a language expert. The evaluation includes about two public hours and five private hours per language; the private, speaker-disjoint set is withheld from distribution to limit leaderboard contamination.
Microsoft leads transcription on the public set
David AI tested eight commercial systems and six open-weight models on the tasks each system supports. On the public evaluation set, Microsoft AI’s MAI-Transcribe-2 had the lowest transcription error rate in 18 of 21 languages and stayed below 10% error in 16. ElevenLabs’ Scribe v2 led the other three languages, including English, and reached below 10% error in 10 languages.
The company cautions that scores should be compared primarily within each language. It uses word error rate for most languages, but character error rate for Mandarin, Japanese, Korean and Thai; differences in writing systems and word boundaries mean the same score can represent different amounts of error across languages. David AI’s results put the broadest system-to-system spread in Bengali, Hindi, Marathi, Tamil and Telugu.
Separate diarization models fare better
For diarization, dedicated models outperformed every system that combines transcription and speaker labeling. The dedicated systems recorded pooled diarization error rates of 18.8% to 25.8%, compared with 32.8% to 44.3% for combined systems. NVIDIA Nemotron 3 Diarization led in nearly all languages in David AI’s comparison.
The task became harder when both speakers spoke at once. David AI reports the highest diarization errors on high-overlap recordings for every system it tested. Its high-overlap tier, however, contains only 15 recordings, a small sample the company flags in its analysis.
“Across both speech recognition and diarization tasks, every system makes the most errors when speakers talk over each other.”
David AI
Access comes with limits
David AI is releasing the evaluation code, inference adapters, scoring tools and language-specific normalizers under the MIT License. The public audio, transcripts and word-level timestamps are available for non-commercial research under a download agreement; the private test set is not distributed. Commercial use of the dataset requires a separate license from the company.
The benchmark has boundaries that matter when interpreting its rankings. It focuses on two-speaker conversations recorded with separate microphones, which can provide diarization systems with cues unavailable in single-microphone recordings or larger group discussions. David AI also notes that many lower-resource languages are not covered, and that language-specific results reflect differences in speakers and recording conditions as well as language itself.