# Artificial Analysis ranks speaker attribution in new benchmark

Published 2026-11-14 · AI.info Team · AI.info

> The Diarization Index measures how often speech-to-text APIs assign words to the wrong speaker or miss them. Artificial Analysis says NVIDIA's Nemotron 3.5 Transcribe led its initial results.

Speech-to-text APIs can now be compared on how often they attribute words to the right person in a new benchmark from [Artificial Analysis](https://x.com/ArtificialAnlys/status/2121351856005984583). [NVIDIA](https://ai.info/companies/nvidia)'s Nemotron 3.5 Transcribe led the initial results with a 3.3% error rate, followed by Voxtral Small Transcribe 3 at 7.5%, the company said in its November 13 announcement.

The Diarization Index evaluates speaker attribution across 97 English conversations involving two to eight people. It sits alongside the company's existing comparisons of transcription accuracy, speed and price on its non-streaming speech-to-text leaderboard.

## How the score works

The index uses word diarization error rate, or WDER: the number of scored reference words assigned to the wrong speaker or missed, divided by the total number of scored reference words. A lower score is better.

The conversations are divided into bands with two, three to four, and five to eight speakers. The headline index is the equal mean of the error rates for those three bands, rather than a single pooled rate across all conversations.

The test calls each API with diarization enabled without telling it how many people are speaking. The conversations total about 6.8 hours.

## Results by dataset

The leaderboard also breaks out results for AA-Diarization, Artificial Analysis's held-out English conversations, which include real and synthetic speech. A separate view covers the NOTSOFAR-1 meetings included in the index set.

The NOTSOFAR-1 result is limited to meetings in the index set, rather than the separate shared-room meetings. Unlike the headline index, each dataset result pools wrong-speaker and missed words across its scored reference words without weighting by speaker band.

The ranking contained 23 APIs, with Mistral's Voxtral Small Transcribe 3 second. AssemblyAI's Universal and [Meta](https://ai.info/companies/meta)'s Muse Voice Transcribe both scored 8.6%, followed by [Microsoft](https://ai.info/companies/microsoft)'s MAI-Transcribe-2 at 8.8%, according to IA Compa's summary of the leaderboard.

## Alongside transcription tests

Diarization measures a different part of a transcription system's output from word error rate. The latter compares the words a model produces against a reference transcript, counting substitutions, insertions and deletions.

Separately, the existing Artificial Analysis Word Error Rate Index uses three datasets, weighted by audio duration. Its non-streaming leaderboard also provides comparisons of diarization scores against price and speed, as well as views of diarization speed and price.

The price measure on the leaderboard is an estimated cost in USD per 1,000 minutes of audio, normalized across providers' billing models. The speed factor measures audio-file seconds transcribed per second of processing time; the leaderboard says its reported values are medians across benchmark trials from the last seven days.

## Sources

1. Original source · Social media post [Artificial Analysis · page 1](https://x.com/ArtificialAnlys/status/2121351856005984583) x.com

Other pages read

2. [Artificial Analysis · Non streaming](https://artificialanalysis.ai/speech-to-text/non-streaming) artificialanalysis.ai
3. [Artificial Analysis · Speech to text](https://artificialanalysis.ai/methodology/speech-to-text) artificialanalysis.ai
4. [IA Compa](https://iacompa.com/news/artificial-analysis-diarization-index-speaker-diarization-benchmark) iacompa.com

## Related stories

- [Google launches benchmark for visual knowledge-work agents](https://ai.info/news/google-launches-visual-knowledge-work-agent-benchmark)
- [Artificial Analysis Puts Frontier Models on a Real Robot Arm](https://ai.info/news/artificial-analysis-aa-robotics-real-robot-arm-benchmark)
- [Meshy says print-preparation tool reduced defects in 3D test](https://ai.info/news/meshy-ai-3d-printability-benchmark)
- [LlamaIndex proposes severity-based scoring for document parsing](https://ai.info/news/llamaindex-parsebench-2-document-parsing-severity)
- [TuringSpace test finds spatial action gaps in leading AI models](https://ai.info/news/turingspace-spatial-action-ai-benchmark)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/artificial-analysis-speaker-diarization-index
