Skip to content
AI.info

The Pulse

Artificial Analysis ranks speaker attribution in new benchmark

The Diarization Index measures how often speech-to-text APIs assign words to the wrong speaker or miss them. Artificial Analysis says NVIDIA's Nemotron 3.5 Transcribe led its initial results.

Artificial Analysis ranks speaker attribution in new benchmark

Speech-to-text APIs can now be compared on how often they attribute words to the right person in a new benchmark from Artificial Analysis. NVIDIA's Nemotron 3.5 Transcribe led the initial results with a 3.3% error rate, followed by Voxtral Small Transcribe 3 at 7.5%, the company said in its November 13 announcement.

The Diarization Index evaluates speaker attribution across 97 English conversations involving two to eight people. It sits alongside the company's existing comparisons of transcription accuracy, speed and price on its non-streaming speech-to-text leaderboard.

How the score works

The index uses word diarization error rate, or WDER: the number of scored reference words assigned to the wrong speaker or missed, divided by the total number of scored reference words. A lower score is better.

The conversations are divided into bands with two, three to four, and five to eight speakers. The headline index is the equal mean of the error rates for those three bands, rather than a single pooled rate across all conversations.

The test calls each API with diarization enabled without telling it how many people are speaking. The conversations total about 6.8 hours.

Results by dataset

The leaderboard also breaks out results for AA-Diarization, Artificial Analysis's held-out English conversations, which include real and synthetic speech. A separate view covers the NOTSOFAR-1 meetings included in the index set.

The NOTSOFAR-1 result is limited to meetings in the index set, rather than the separate shared-room meetings. Unlike the headline index, each dataset result pools wrong-speaker and missed words across its scored reference words without weighting by speaker band.

The ranking contained 23 APIs, with Mistral's Voxtral Small Transcribe 3 second. AssemblyAI's Universal and Meta's Muse Voice Transcribe both scored 8.6%, followed by Microsoft's MAI-Transcribe-2 at 8.8%, according to IA Compa's summary of the leaderboard.

Alongside transcription tests

Diarization measures a different part of a transcription system's output from word error rate. The latter compares the words a model produces against a reference transcript, counting substitutions, insertions and deletions.

Separately, the existing Artificial Analysis Word Error Rate Index uses three datasets, weighted by audio duration. Its non-streaming leaderboard also provides comparisons of diarization scores against price and speed, as well as views of diarization speed and price.

The price measure on the leaderboard is an estimated cost in USD per 1,000 minutes of audio, normalized across providers' billing models. The speed factor measures audio-file seconds transcribed per second of processing time; the leaderboard says its reported values are medians across benchmark trials from the last seven days.

Sources

Related stories

Explore

More articles