MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models

AI summary

Overview

Research area: Speech and language processing — specifically evaluation benchmarks for Full-Duplex Speech Language Models (FD-SLMs), the class of models that can listen and speak simultaneously.

Technical level: Intermediate. The paper is accessible to anyone familiar with dialogue systems and speech models, but the turn-segmentation algorithm and evaluation design assume some familiarity with speech pipeline concepts (VAD, ASR timestamps, teacher forcing).

Scope: The paper introduces MTR-DuplexBench, a benchmark that evaluates full-duplex speech models over multi-round conversations across four dimensions, using a new method for slicing continuous overlapping audio into discrete turns.

What This Paper Is About

Full-duplex speech models can talk and listen at the same time, enabling interruptions and backchannels, but existing benchmarks only test them in a single round of interaction. Testing them over many rounds is hard because full-duplex audio has no clean "turn boundaries" (both speakers overlap), and the model's own earlier errors cause its context to drift away from the ground-truth conversation. The paper builds a benchmark that solves both problems and then evaluates models on conversation behaviour, dialogue quality, instruction following, and safety across up to ten rounds.

Key Contributions

1. MTR-DuplexBench, a benchmark for multi-round evaluation of FD-SLMs that covers four evaluation dimensions: conversational features, dialogue quality, instruction following, and safety. It is the first full-duplex benchmark to combine multi-round dialogue, turn-by-turn evaluation, and all four dimension categories.

2. A full-duplex turn segmentation methodology that converts continuous, overlapping two-channel dialogue into discrete user turns and corresponding assistant response windows. It combines Whisper-timestamped transcription, Silero voice activity detection, and GPT-4o-based segmentation, then stabilizes results with six repeated runs, majority voting with a 30% time-overlap clustering rule, and a final overlap-resolution pass.

3. A context-consistency fix for dialogue quality evaluation: when evaluating turn i, all preceding assistant turns are populated with ground-truth speech (teacher forcing) so the model is never placed in a conversation state that would never occur in reality. The assistant's response window runs from the start of the current user turn to the end of the next user turn, with the next user turn muted.

4. An empirical study of four baselines — Moshi (end-to-end FD-SLM), Freeze-Omni (cascaded FD-SLM), VocalNet (half-duplex SLM), and an ASR+LLM+TTS cascade — showing that current FD-SLMs degrade across rounds and dimensions.

Main Findings

• Performance drops steadily as rounds accumulate. Across all four tested conversational features (smooth turn-taking, interruption, pause handling, background speech), success rates fall as conversations go from 1 round to 10. Moshi's smooth turn-taking drops from 73.0% to 57.4%; Freeze-Omni's falls from 69.0% to 36.4%.

• Latency also grows with rounds. Response latency rises monotonically in every feature category for most models, meaning the conversation feels progressively less natural. The authors argue that the commonly reported "first-package latency" is insufficient and that multi-round latency should be standard.

• Combining features is harder than handling one at a time. Stacking smooth turn-taking with interruption, pause handling, and background speech causes further degradation. Background speech is the hardest single feature: Moshi drops to 25.7% at round 10, while Freeze-Omni, VocalNet, and the cascade all fall to roughly 0–1% because any detected input halts a half-duplex design.

• Half-duplex and cascaded systems score well on most features but fail background speech completely. Their design guarantees a full response after the user's utterance ends, which is an advantage for turn-taking and pause handling, but makes them unable to ignore irrelevant background audio.

• The cascaded ASR+LLM+TTS approach has extreme latency — roughly 9 to 12 seconds per turn, an order of magnitude worse than the speech-native models.

• Dialogue quality trades off against full-duplex capability. On rescaled GPT-scores (0–5), VocalNet (half-duplex) scores 3.96, Freeze-Omni (cascaded FD-SLM) 3.48, and Moshi (end-to-end FD-SLM) 3.13. The pure cascade scores lowest at 2.19, largely because its latency prevents it from finishing responses — its average response is only 10 words versus 20+ for the others.

• Instruction following degrades only for Moshi. Moshi falls from 68.0% at round 1 to 41.9% by round 10. Freeze-Omni, VocalNet, and the cascade stay roughly flat across rounds. Moshi also scores lowest even in round 1.

• Safety is the easiest capability to maintain. All baselines keep refusal rates at or above 90% across every round and condition, with Freeze-Omni near 100%.

• Interruptions do not meaningfully hurt instruction following or safety. When every round is changed from smooth turn-taking to interruption, all models show nearly identical success and refusal rates, indicating robustness to repeated disruption.

Methodology in Plain English

The researchers start with real or synthetic two-channel audio (one channel per speaker) and need to convert it into a sequence of labelled rounds. First, they transcribe both channels and run voice activity detection to get noisy, fragmented time segments. They feed these segments to GPT-4o, which decides where each user turn actually begins and ends. Because GPT-4o gives slightly different answers each time, they repeat this six times and keep turns that overlap in time by at least 30% across runs, taking the median start and end times. A final pass merges any leftover overlaps. This produces clean user turns despite the fact that both speakers were talking over each other in the original audio.

For each extracted turn, the model under test is asked to respond, while all earlier assistant turns are filled in with ground-truth speech so the model does not have to reason about its own earlier mistakes. This is the teacher-forcing step that keeps the conversation on a realistic trajectory. Whether the response window ends is bounded by the start of the next user turn, with that next turn muted so the model can finish speaking.

Evaluation then differs by dimension. Conversational features and instruction following/safety use synthetic or independent multi-turn data where the model builds autoregressively on its own past outputs. Dialogue quality uses natural human conversations from the Candor dataset and requires turn segmentation plus teacher forcing. Metrics are a mix of rule-based success criteria (turn-taking, interruption, pause handling), GPT-4o binary judgements (instruction following, safety), and GPT-4o 0–5 scoring for dialogue quality, rescaled because GPT-4o is conservative and gives ground-truth dialogues only 2.97 out of 5.

The baselines are Moshi and Freeze-Omni (the only two open-source FD-SLMs), VocalNet as a half-duplex comparison, and a pipeline combining SenseVoice, GPT-4o, and ChatTTS as a cascaded comparison. The half-duplex and cascaded systems are given the assumption that they stop speaking the moment user speech is detected.

Why This Matters

Impact on research. The benchmark establishes that single-round evaluation is misleading for full-duplex models: a model that looks strong in round one may be substantially weaker by round ten. It also makes turn-level measurement on continuous overlapping audio feasible, which previously required either manual annotation or accepting blurred boundaries. The teacher-forcing design gives a reproducible way to compare models fairly when their own outputs would otherwise push them into divergent contexts.

Real-world applications:

• Voice assistants and smart speakers, where users interrupt, pause mid-sentence, and hold conversations lasting many turns rather than issuing a single command.
• Customer service automation, where a bot must keep following instructions and staying safe even after the caller has interrupted it repeatedly.
• Accessibility tools such as conversational aids for users with speech or motor impairments, where latency and turn-taking accuracy directly affect usability.
• In-car and hands-free interfaces, where background speech (passengers, radio, road noise) is the dominant failure mode — exactly the feature where current FD-SLMs collapse to near-zero success.
• Real-time translation and live interpretation, where overlapping speech and long multi-turn context are the norm.

Industry relevance. The results give engineering teams concrete tradeoffs. End-to-end FD-SLMs deliver the fastest, most natural timing but the weakest dialogue semantics and instruction following. Cascaded systems are more semantically reliable but incur roughly 10 seconds of latency, which destroys the conversational experience. No current architecture wins on all axes, so the benchmark provides a scorecard for deciding which compromises are acceptable for a given product.

Future Directions

• Reducing multi-round latency. The finding that latency grows with conversation length is unexplained and unaddressed. Understanding whether this comes from context accumulation, state drift, or inference overhead is a clear next problem.

• Unifying the evaluation paradigm. The paper acknowledges that dialogue quality uses teacher forcing while the other dimensions use unguided autoregressive inference. A single framework that handles natural data without teacher forcing would remove the gap between the evaluation setup and true model behaviour.

• Broadening language and data coverage. The benchmark is English-only and mixes natural with synthetic audio. Multilingual and cross-lingual FD-SLM evaluation, plus more diverse natural full-duplex corpora, remain open.

• Expanding the feature set. The design inherits its feature list from prior benchmarks, so it omits things like user backchannels. Adding more interaction dynamics would give a fuller picture.

• Improving end-to-end instruction following. Moshi is the only model whose instruction-following ability collapses over rounds, and it is also the only truly end-to-end FD-SLM tested. Closing this gap appears to be the central open challenge for end-to-end architectures.

Target Audience

Researchers and engineers working on speech language models, spoken dialogue systems, and real-time conversational AI. It is most useful to those building or benchmarking full-duplex models, and to evaluation-focused researchers interested in how to measure continuous, overlapping speech at the turn level. Product and infrastructure teams choosing between end-to-end and cascaded voice architectures will also find the tradeoff data directly applicable. Readers should have basic familiarity with ASR, voice activity detection, and dialogue evaluation to get the most out of the methodology section.

Authors’ abstract

Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Also, existing benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark designed for a comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our benchmark. Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench