Overlapped Speech, Meeting Capture, and Multi-Speaker Attribution

Meeting capture is not one task but five, run in sequence: detect that someone is speaking, recognize that more than one person is, estimate where the turns fall, separate or transcribe the simultaneous voices, and bind the resulting words to the right participant. Each stage fails in its own way. Each inherits whatever the stage before it got wrong.

That inheritance has been priced. The CHiME-6 challenge ran the same recognizer twice in 2020. Given oracle segmentation and BeamformIt, the Track 1 baseline scored 61.2% WER on the evaluation set. In Track 2 the identical acoustic and language models were fed by automatic diarization instead. They scored 77.9%, with a baseline diarization error rate of 62.0–68.2%. Nothing about the recognition had changed. Only who decided where the turns fell and whose they were. The challenge organizers say so outright: “Therefore, by comparing Tracks 1 and 2 with BeamformIt, we can observe that the main degradation (around 15% absolute) comes from speaker diarization.” The same challenge introduced cpWER, the concatenated minimum-permutation word error rate, and recorded the best Track 2 system at 44.5%.

The last stage is the odd one out. It is the only one that makes a claim about a person. The first four describe a signal, and can be checked against the signal. Binding a sentence to a named participant is a claim about who was in the room and what they said. The evidence points at somebody. It never settles the matter.

Where that binding happens — after segmentation, after separation, or jointly with recognition — is a real argument. It is worth having before the architecture is fixed, because the order decides whose errors everyone else has to live with. What no ordering can change is the shape of the timeline underneath. A timeline that allows one active speaker at a time has no way to represent true overlap. So when two people talk at once, it does not report a problem. It chooses.

Such a reference makes one kind of error visible. A board meeting system, during a passage of crosstalk, produced an accurate sentence and attached it to the louder speaker. Every word was right. What changed was who had objected. In the minutes of a board meeting that is more or less the whole content of the record. Word error rate barely moved.

That is the trap, and it is a reporting trap as much as a modeling one. A review that reports only the diarization error, with a note on how overlap was counted, never has to confront the moment the system picked one speaker out of two who were talking at once. That moment is when the record was decided. And it is not an edge case: 20% of the 52 speakers Çetin and Shriberg analysed were overlapped more than 30% of the time.

What the boardroom lost was accountability: a decision recorded against the wrong name. Move the identical error into another room and the currency it is paid in changes with it. The same misattribution is counted in different units depending on where it lands. Each setting has to be argued on its own terms, not by analogy with the board meeting.

Naming those four failures precisely depends on four words that are easy to swap for one another. A swap changes what has to be proved, in what unit, and who gets to decide. Speaker-attributed WER is not word error rate with a footnote, and crosstalk is not another name for overlap detection. Turn is the unit that carries both.

A benchmark makes the distinction concrete. Microsoft released NOTSOFAR-1 in 2024: 315 real meetings averaging 6 minutes, recorded across 30 conference rooms with 4–8 attendees and 35 unique speakers. Alongside it came a 1000-hour simulated training set, built from 15,000 real acoustic transfer functions. Systems were scored with the speaker-attributed tcpWER, and with a deliberately separate speaker-agnostic word-error metric. The reason for keeping two is stated outright: “However, speaker attributed metrics such as tcpWER are highly sensitive to speaker diarization errors: an incorrectly attributed segment will be penalized even if the words within it are accurately recognized.”

That sentence is the boardroom failure written as a scoring rule.

Because the turn carries the words and the name together, where a system forms its turns decides a great deal. A diarization-first pipeline segments and clusters before or around ASR, so the turns exist before the words do. A separation-first pipeline creates source streams before recognition and attribution. A joint multi-speaker model optimizes several outputs together, with stronger modeling and data demands. Each commits to a speaker at a different moment. So each has to be checked on its own terms, and a good result from one is not evidence about another.

The separation-first case has a measured curve rather than a description. LibriCSS is 10 hours of real far-field replay audio in 10 sessions. Each session splits into six 10-minute mini-sessions, at overlap ratios of 0%, 10%, 20%, 30% and 40%, with eight speakers each. Without separation, utterance-wise WER rises from 11.8% at 0% overlap to 43.3% at 40%. Seven-channel MVDR continuous speech separation cuts that 40%-overlap figure to 21.7%. Just as importantly, it does not damage the easy end: the 0%-overlap case improves to 8.4%. That second number is the one to insist on. A front-end that halves the error under heavy overlap while degrading the overlap-free stretches has moved the failure rather than removed it. Natural conversation contains both kinds of component in the same recording.

The three columns are then judged on different things. Diarization-first is judged on diarization error with the treatment of overlap stated, and watched for forcing one active speaker during interruptions. Separation-first is judged on speaker-attributed word error rate, and watched for using names mentioned in speech as identity labels. The joint model is judged on overlap detection and source-count accuracy, and watched for evaluating words without speaker attribution. Each is best used when its assumptions are written down.

Return to the number that barely moved. Word error rate sat still through the boardroom misattribution. A diarization error quoted on its own can sit just as still — or move a long way without the system changing at all.

One diarization system, one set of recordings, two very different numbers. A powerset model with domain adaptation scores 18.0% DER on AMI headset-mix and 22.9% on AMI array-1-channel-1 with no forgiveness collar. On the identical audio, once a 250 ms collar is applied, it scores 11.2% and 15.3%. Plaquet and Bredin published both in 2023, and their table caption says why: “No forgiveness collar is used for evaluation, except for numbers in italics with a grey background computed with a 250ms forgiveness collar (to allow comparison with some results reported in the literature).” 18.0% and 11.2% are the same system on the same recordings. Only the scoring rule differs. Their powerset gain over multi-label, incidentally, came mostly from overlapped speech, cutting missed detection from 13.1% to 9.9% on average.

Stating the rule is not a courtesy. A challenge can require it. The Third DIHARD Challenge fixed the choice in writing in 2020, and reversed prior practice: “Contrary to practice in the NIST RT evaluations, NO forgiveness collar will be applied to the reference segments prior to scoring and overlapping speech WILL be evaluated.” The same rule was then applied to 499 valid submissions from 21 teams, scored with dscore v1.0.1.

Four numbers are needed, because four different questions are being asked. Take away the other three and the diarization error is carrying the release claim on its own.

A meeting error taxonomy is the instrument that keeps those four numbers honest, and it is built to be argued with. Its job is to give a reader a way to challenge what the system does when two people talk at once and only one of them can be named.

Three notes are enough to begin. Write down what your activity and overlap detection assumes about how people talk in this particular room. Check that assumption against a measured rate rather than an impression: overlap runs at 7.1% in AMI's elicited scenario meetings against 13.0% in real ICSI ones, as Çetin and Shriberg found. Then write one meeting that would break the assumption. Then write what a bad attribution — the boardroom kind, right words and wrong name — makes you change in the binding and review steps. A taxonomy that changes nothing downstream is just a list.

The four moves it encodes are these. Separate the error axes: missed speech, extra speech, wrong words, wrong speaker, wrong timing, tracked apart. The powerset gain Plaquet and Bredin measured showed up as missed detection falling from 13.1% to 9.9%, and would have been invisible inside an aggregate. Create hard scenarios: interruption, laughter, side speech, remote participants, speaker movement. LibriCSS built its hard scenarios by construction, six mini-sessions per session at 0% through 40% overlap. Expose uncertainty: show low-confidence attribution and allow segment reassignment. Audit the summaries: verify that disputed speakers and statements are not silently resolved.