SpeakerSleuth: Can Large Audio-Language Models Judge Speaker Consistency across Multi-turn Dialogues?

AI summary

Overview

Research area: Speech and audio-language model evaluation (natural language processing / speech synthesis quality assessment).

Technical level: Intermediate. The task definitions and motivation are accessible without a speech-processing background, though the results tables involve metrics like Balanced Accuracy, NDCG, and Exact Match.

Scope: This paper introduces SpeakerSleuth, a benchmark of 1,818 human-verified instances designed to test whether Large Audio-Language Models (LALMs) can judge whether a speaker's identity stays consistent across the turns of a multi-turn dialogue.

What This Paper Is About

Speech synthesis systems can now generate natural-sounding dialogue, but they sometimes drift in voice identity across turns — a problem that only becomes audible when several utterances are heard together, not in any single clip. Existing evaluation methods compare two utterances at a time with manually set thresholds, and no unified benchmark existed to test whether LALMs can judge speaker consistency across a whole dialogue. The paper builds such a benchmark and uses it to measure how well twelve LALMs and six speaker-embedding methods detect, locate, and rank speaker inconsistencies.

Key Contributions

1. SpeakerSleuth, a new benchmark. The authors present what they describe as the first benchmark for multi-turn speaker consistency evaluation, containing 1,818 human-verified instances drawn from 606 unique dialogues and 197 speakers, totaling 10.2 hours of audio across four source datasets.

2. Three decomposed tasks. Rather than a single score, the benchmark splits speaker consistency into Detection (is the dialogue consistent?), Localization (which turn is the problem?), and Discrimination (which of three candidates best matches the target speaker?), mirroring a practical generate-detect-regenerate workflow.

3. A controlled scenario design. Each dialogue is presented in three versions with identical dialogue content: S1 (Fully Consistent), S2 (Gender Switch), and S3 (Similar Speaker, where the substituted voice is the acoustically closest speaker by ECAPA-TDNN cosine similarity). Holding the text constant isolates acoustic difficulty.

4. A systematic comparison of 12 LALMs against 6 embedding methods, revealing that models have unstable internal thresholds for detection, struggle to localize problematic turns, and perform substantially better at relative discrimination than absolute judgment.

Main Findings

• Detection is unreliable across the board. Most LALMs score below 60% balanced accuracy. The best LALM, Gemini-2.5-Pro, reaches 64.7% but only 39.3% on the hardest S3 scenario. The best embedding method, Pairwise (WavLM), reaches 64.9%.

• Models cluster at opposite extremes. Some models (MiniCPM-o-2.6, Audio-Flamingo-3) almost always predict "consistent" and miss even obvious speaker changes, while others (Gemini-2.5-Flash-Lite, Qwen2.5-Omni-7B) show the opposite bias. Audio-Flamingo-3 achieves 99.2% on S1 but 1.3% and 1.2% on S2 and S3.

• Localization is harder than detection. Even methods with strong detection do not proportionally improve at pinpointing the inconsistent turn; the best balanced F1 is 61.7% (Pairwise ECAPA). Only Gemini-2.5-Pro (S2/S3 F1 of 62.5%/47.5%) and Gemini-2.5-Flash (62.3%/45.0%) keep meaningful precision alongside high recall, and both drop from S2 to S3, showing they do respond to acoustic difficulty.

• Textual context hurts rather than helps. When other interlocutors' turns are supplied as text, most models become far more likely to call a dialogue consistent. GPT-4o-audio gains 20.5 points on S1 but loses 26.5 and 24.5 points on S2 and S3; Gemini-2.5-Flash-Lite gains 52.6 points on S1 while losing 67.0 and 66.0 points on S2 and S3. Gemini-2.5-Pro degrades across all three scenarios.

• Reference audio anchors judgments. Removing the reference sample pushes several models toward lenient "consistent" verdicts. Qwen2.5-Omni-7B rises 66.5 points on S1 while dropping 69.5 and 67.8 points on S2 and S3.

• Discrimination is a relative strength. Gemini-2.5-Pro reaches 81.5% classification accuracy, 92.6% NDCG@2, and 71.5% Exact Match, ahead of Gemini-2.5-Flash (75.6%, 88.3%, 61.6%) and Qwen3-Omni-30B-A3B (60.4%, 73.2%, 36.8%). Embedding methods reach near-perfect classification (93–99%) and high NDCG@2 (91–93%) but their Exact Match drops to 55–59% — the reverse tradeoff from Gemini-2.5-Pro.

• Longer clips help. Performance on S2/S3 rises monotonically with clip duration quartile for the top three LALMs by discrimination accuracy, indicating models need sufficient acoustic evidence per clip.

• Results hold at 10 turns. A 10-turn dataset of 759 instances from 253 unique dialogues shows an average drop of 1.2% Detection, 4.2% Localization, and 3.1% Discrimination, with model rankings largely preserved.

• Voice-cloning quality ranking remains hard. On a VC Quality Ranking task with four candidates per sample (source, OpenVoice, YourTTS, CosyVoice3) and human rankings from three annotators over 485 samples (inter-annotator Kendall's W = 0.860), Gemini-2.5-Pro leads at 48.0% accuracy and 75.1% NDCG@2, but Exact Match is only 16.8%. Random baselines are 25.0% accuracy, 50.4% NDCG@2, and 4.2% Exact Match.

Methodology in Plain English

The authors started with an existing pool of dialogues that already had both audio and transcripts, drawn from four sources: Bazinga (TV shows and movies), AMI (business meetings), Behavior-SD (synthesized dialogues with conversational behaviors), and DailyTalk (studio-quality everyday conversations). This gave 3,683 dialogues covering 1,358 unique speakers.

From that pool they selected segments where one target speaker appears repeatedly, keeping exactly 5 target-speaker turns and capping the total number of turns at 20 so that a target speaker's turns do not drift too far apart.

For each selected dialogue, they produced three matched versions. The original audio is the S1 fully consistent case. For S2, they randomly picked one turn and applied voice conversion to an opposite-gender voice. For S3, they converted it to the acoustically most similar other speaker, chosen by comparing ECAPA-TDNN embeddings with cosine similarity. The primary conversion tool was FreeVC, with an extended benchmark built using CosyVoice3, OpenVoice, and YourTTS to check robustness.

Quality control used two gates: Qwen3-32B filtered out segments that read unnaturally when taken out of context, and three expert annotators checked audio clarity, absence of noise or artifacts, and the naturalness of converted turns. Only instances passing both were kept, leaving 606 dialogues and 1,818 instances.

Evaluation gives a model all of the target speaker's turns at once, plus a reference audio sample of at least 3 seconds, and asks it to judge consistency. Detection is scored with per-scenario accuracy and a Balanced Accuracy that weights S1 against the average of S2 and S3 equally. Localization is scored with macro-averaged precision, recall, and F1, plus a Balanced F1. Discrimination offers three candidates in randomized order and is scored either as classification accuracy or as ranking with NDCG@1, NDCG@2, and Exact Match. The embedding baselines use pairwise cosine similarity (threshold 0.4), centroid distance (threshold 0.3), or reference comparison (threshold 0.4), each with WavLM and ECAPA-TDNN backbones.

Why This Matters

Impact on research. The paper reframes speaker consistency from a pairwise similarity problem into a three-part capability and shows a dissociation: models that cannot make a stable absolute judgment can still compare acoustic variants well. That points to calibration and multimodal attention balance, rather than raw acoustic representation quality, as the bottleneck. It also supplies a shared benchmark that had been missing for comparing embedding methods and LALMs on equal footing.

Real-world applications:

• Voice assistants and conversational agents that must keep a fixed voice identity across a long session.
• Podcast generation and audiobook voice-overs, where a narrator's timbre must not drift across episodes or chapters.
• Movie and TV dubbing, where multiple characters speak in turn and each needs a stable, distinct voice.
• TTS validation pipelines: detect an inconsistency, regenerate, and pick the best candidate before shipping audio.

Industry relevance. The results carry a direct practical recommendation: providing reference audio is essential for reliable speaker consistency judgment, and supplying dialogue text can actively undermine it. Teams building automated quality gates for speech products should weigh those two design choices carefully rather than assuming richer context always helps.

Future Directions

• Multilingual extension. The benchmark covers only English dialogues; the authors note the synthetic generation pipeline is language-agnostic and can be extended.
• Isolating acoustic factors. The four source datasets vary in noise, reverberation, and recording quality, but the paper does not separate these factors; the pipeline supports adding controlled perturbations via standard audio augmentation.
• Demographic bias analysis. The paper does not analyze performance across accents or age groups, leaving open whether LALMs show demographic biases in consistency judgment.
• Modality integration. The authors call for mechanisms that balance attention allocation during multimodal fusion, so that dialogue context helps rather than overwhelms acoustic evidence.
• Better voice-cloning evaluators. Reliable ranking of cloned outputs against human judgment remains an open problem, given low Exact Match even from the strongest model.

Target Audience

Researchers and engineers working on speech synthesis, speech quality evaluation, and audio-language model development will get the most from this paper. It is also relevant to practitioners designing automated validation for TTS or voice cloning products, and to NLP researchers interested in modality imbalance in multimodal models. Readers do not need deep speech-processing expertise, though familiarity with speaker embeddings and evaluation metrics such as NDCG will help in interpreting the results tables.

Authors’ abstract

Large Audio-Language Models (LALMs) as judges have emerged as a prominent approach for evaluating speech generation quality, yet their ability to assess speaker consistency across multi-turn dialogues remains unexplored. We present \textbf{SpeakerSleuth}, a benchmark evaluating whether LALMs can reliably judge speaker consistency across multi-turn dialogues through three tasks reflecting real-world requirements. We construct 1,818 human-verified evaluation instances across four diverse datasets spanning synthetic and real speech, with controlled acoustic difficulty. Evaluating twelve widely-used LALMs, we find that models struggle to reliably detect acoustic inconsistencies. For instance, given audio samples of the same speaker's turns, some models overpredict inconsistency, whereas others are overly lenient. Models further struggle to identify the exact turns that are problematic. When other interlocutors' turns are provided as textual context, performance degrades dramatically as models prioritize textual coherence over acoustic cues, failing to detect even obvious gender switches for a speaker. On the other hand, models perform substantially better in comparing and ranking acoustic variants, demonstrating inherent acoustic discrimination capabilities. These findings expose a significant bias in LALMs: they tend to prioritize text over acoustics, revealing fundamental modality imbalances that need to be addressed to build reliable audio-language judges. Our code and data are available at https://github.com/holi-lab/SpeakerSleuth.