Speaker Diarization and Overlap Attribution

Diarization error rate reports one thing: how much reference speaker time was attributed to the wrong speaker. It does not report which way the timeline broke. Hervé Bredin said so in one sentence in 2017, presenting pyannote.metrics: “While the diarization error rate provides a convenient way to compare different diarization approaches, it is usually not enough to understand the type of errors commited by the system.”

His remedy was two numbers instead of one. pyannote.metrics reports cluster purity and coverage as a pair. They separate over-segmentation — one speaker split into several — from under-segmentation, several speakers merged into one. One participant cut into three clusters and three participants collapsed into one cluster are opposite failures with opposite fixes. The headline error rate leaves them looking alike. It counts mislabelled time. It says nothing about whether the system made too many identities or too few.

A timeline whose one published number is silent about the shape of its own mistakes. That is what the rest of this lesson works on.

Clustering diarization groups segment embeddings without fixed speaker labels. Each stretch of the timeline is assigned to one cluster, so a passage where two people talk at once is credited to a single speaker. Nothing in the method holds a label steady across a change in the audio either. There were never any labels to hold.

The first of those costs has been measured. In 2019 an end-to-end neural diarization system from Fujita and colleagues reached 12.28% DER on simulated speech mixtures. A conventional clustering-based system produced 28.77% on the same material. Their abstract names the reason: “Besides its end-to-end simplicity, the proposed method also benefits from being able to explicitly handle overlapping speech during training and inference.” That gap is not a tuning difference. It is the difference between a method that can label two active speakers and one whose output has no room to say so.

Attributed diarization starts somewhere else again. It maps anonymous speaker tracks to known participants using additional evidence. Anonymous tracks and named participants are not two grades of the same output. They are answers to different questions. A result from one of these three approaches settles nothing about the other two. A 12.28% DER on mixtures says nothing about whether the right name was attached to the track.

Underneath all three sits one definition. Speaker diarization partitions speech over time by anonymous speaker identity: who spoke when, not who they are. A system gets there by combining activity detection, segmentation, embeddings, clustering or sequence modeling, overlap handling and optional resegmentation. Speaker naming is a separate process that happens afterwards. Diarization does not establish legal identity.

What it does instead is fail in ways the timeline never displays. It can split one person into several clusters — the over-segmentation purity and coverage were built to expose. It can merge different people into one. It can mis-handle overlap, which the primary NIST-style metric was not scoring in the first place. And it can inherit channel or seating cues that hold up only until participants move.

So the question is not how good the model is. It is whether the speech and overlap you detected still support the attribution you make on top of them. That link decides whether a diarization score can back the decision at all.

Attribution means something different in a meeting, a call center, a broadcast archive and a clinical conversation. DIHARD III put numbers on how far apart those settings really are. The 2021 write-up drew on 499 valid system outputs, from 30 organisations forming 21 teams, across 11 domains. Median Track 1 DER came in below 10% for six of those domains. For clinical interviews and telephone speech it sat between 10% and 20%.

Then the floor drops. From the results section: “For the final three domains – meeting speech, web videos, and restaurant – performance ranges awful with median track 1 DER ranging from 35% to 45%.”

Same task, same metric, same year: a sub-10% problem in six settings and a 35–45% problem in three. A diarization result carries its domain with it. The core evaluation set of that challenge is itself 8.75% overlapped speech.

Those unmarked turns are produced in a particular order, and the order is what makes them invisible. The work creates speaker evidence first, then detects speech and overlap, then attributes cautiously.

The first step is the one that goes when a deadline arrives. Embeddings need a window long enough to be worth extracting from, and shortening that window is the cheapest cut available. Drop it and the detection step is working from evidence nobody checked. The attribution step never tests the assumption either. Being cautious about the marks you were handed is not the same as asking about the ones that were never made.

Inferring speaker structure is where the count gets decided, and deciding it in advance has a measured price. In 2020 Horiguchi and colleagues attached encoder-decoder based attractors (EDA) to an end-to-end model, so that the number of speakers is produced rather than assumed. On CALLHOME with an unknown number of speakers the result was 15.29% DER. The x-vector-based clustering method gave 19.43%. On simulated two-speaker mixtures EDA scored 2.69%, where vanilla SA-EEND — the fixed-speaker-count end-to-end baseline, not a clustering system — gave 4.56%. Nearly two points on the two-speaker case, where the count was not even in doubt. That is what an assumption about speaker count costs when it is baked into the architecture rather than estimated.

An audit runs that order backwards. The timeline is the artefact under audit. It should be detailed enough that another team can check whether overlapped speech was scored or quietly dropped.

The first move is to stop auditing against a single figure. pyannote.metrics gives the reading that tells you which failure you have: “Over-segmented results (e.g. too many speaker clusters) tend to lead to high purity and low coverage, while under-segmented results (e.g. when two speakers are merged into one large cluster) lead to low purity and higher coverage.” High purity with low coverage is one person cut into several tracks. Low purity with higher coverage is several people wearing one label. Separating missed speech, false alarm, confusion, split, merge and overlap errors is the same discipline applied column by column.

Three notes are then enough, and they follow the pipeline. First, what does your speech and overlap detection assume? Then one case that breaks it: a participant who changes microphones halfway through the meeting, or the 19% of AMI evaluation speech that arrives with two people talking. Then what the attribution step does when that case appears — merge the tracks, mark them uncertain, or carry on naming speakers who are not there.

What an audit reads by hand, an evaluation has to hold as two numbers at once. DIHARD II showed how far apart two metrics on one system can sit. Jaccard error rate was introduced there in 2019 and defined this way: “An optimal mapping between speakers in the reference diarization and speakers in the system diarization is determined and for each pair the Jaccard index of their segmentations is computed. JER is defined as 1 minus the average of these scores, expressed as a percentage.” On the identical baseline output, Track 1 scored 25.99% DER and 59.51% JER on the evaluation set. Track 2 scored 50.12% DER against 72.1% JER.

Nothing changed between those pairs except what was being counted. DER weights time. A system that handles the long-talking speakers well can look strong while short-turn speakers are being lost. JER averages per speaker, and refuses to let that trade go unrecorded. Report one and you have chosen which speakers matter without saying so.

The set of metrics should also mark the point where overlap has been left out of the reference, so the timeline is flagged rather than trusted.