Spoken DialogSum: An Emotion-Rich Conversational Dataset for Spoken Dialogue Summarization

AI summary

Overview

Research area: Spoken dialogue summarization, emotion-aware multimodal modeling, and speech-language dataset construction (Natural Language Processing / Speech Processing).

Technical level: Intermediate — the paper assumes familiarity with ASR, TTS, ROUGE metrics, and Audio-LLM architectures, but its core ideas and pipeline are explained clearly enough for readers with a general machine learning background.

One-sentence scope: The paper introduces Spoken DialogSum, a 13,460-dialogue synthetic spoken corpus that pairs multi-speaker conversational audio with factual summaries, emotion-rich summaries, and utterance-level age, gender, emotion, pitch, and speaking-rate labels, then benchmarks text-only, cascaded, and end-to-end audio models on three tasks.

What This Paper Is About

Dialogue summarization datasets such as DialogSum and SAMSum contain only written transcripts, while spontaneous speech corpora such as Switchboard and MELD contain real vocal signals but no summaries. This leaves a gap: no resource links raw conversational audio, factual summaries, emotion-rich summaries, and paralinguistic labels in one place, which blocks progress on emotion-aware spoken dialogue summarization.

The authors address this by converting DialogSum scripts into naturalistic spoken conversations and synthesizing expressive multi-speaker audio for them, producing a corpus designed to test whether models can fuse what is said with how it is said.

Key Contributions

1. A new corpus linking speech, summaries, and paralinguistics. Spoken DialogSum contains 13,460 multi-speaker dialogues and 251,575 utterances (159.87 hours total), each paired with both a factual summary and an emotion-focused summary, plus utterance-level labels for speaker age, gender, emotion, pitch, and speaking rate. The authors describe it as the first corpus to pair raw multi-speaker audio with both factual and emotion-rich summaries while also providing utterance-level speaker-attribute labels.

2. A three-stage data generation pipeline. Scripted DialogSum dialogues are rewritten with Switchboard-style fillers and hesitations by LLAMA3.3 70B, back-channels are inserted at natural pause points, and each utterance is tagged with one of eight emotions plus discretized pitch and speaking-rate values by GPT-4o-mini.

3. An expressive TTS synthesis framework with timing-driven overlap. Zonos-hybrid synthesizes speech conditioned on speaker prompts drawn from a bank of 20,385 voices derived from GigaSpeech, with interruption and back-channel placement governed by timing statistics from the CANDOR corpus.

4. A three-task benchmark suite. The paper defines factual summarization (semantic only), age/gender/emotion classification (paralinguistic only), and emotion-rich summarization (semantic × paralinguistic), and evaluates seven baseline systems across them.

Main Findings

• Writing quality improves over the source corpora. GPT-4o-mini ratings on Oral Naturalness, Conversational Flow, and Topical Coherence give Spoken DialogSum an average of 4.48 (Nat. 4.81, Flo. 4.15, Coh. 4.49), compared with DialogSum at 4.19 (3.86 / 4.13 / 4.59) and Switchboard at 4.02 (4.25 / 3.71 / 4.11). Topical Coherence drops slightly versus the original DialogSum (4.59 to 4.49), which the authors attribute to back-channels fragmenting topical continuity.

• Text-only models win on factual summarization. Transcription + LLaMA 2 reaches ROUGE-L 21.8 and BERTScore F1 87.6, and Whisper + LLaMA 2 reaches ROUGE-L 22.0 / 87.0, topping the dialect summarization leaderboard. SALMONN-7B is lowest at ROUGE-L 13.5.

• The ranking reverses on emotion-rich summarization. SALMONN-13B leads with ROUGE-1 35.9, ROUGE-2 13.3, ROUGE-L 30.8, and BERTScore F1 91.5, with WavLLM close behind (33.4 / 8.8 / 27.8 / 91.1). The cascaded Whisper + LLaMA 2 pipeline falls to ROUGE-L 21.6.

• Audio-LLMs beat cascaded pipelines on emotional summaries. The abstract and conclusion report a 28% relative ROUGE-L gain for an Audio-LLM over the cascaded ASR-LLM system; the introduction states 29% for the same comparison.

• Age and gender are recoverable from the audio. A wav2vec 2.0-based classifier reaches 66.3% accuracy / 65.2% F1 on age and 95.4% accuracy / 95.4% F1 on gender, close to the 67.7% accuracy / 80.7% F1 (age) and 95.7% / 95.7% (gender) on EMODB. Audio-LLMs perform far worse: WavLLM gets 31.4% / 29.0% on age and 59.7% / 59.1% on gender; Qwen-Audio gets 48.8% / 45.0% and 51.0% / 34.5%.

• Emotion recognition trends track human-labeled benchmarks. In a 4-class setup, LTU-AS reaches 47.75% accuracy on EmoSum versus 49.12% on IEMOCAP, while WavLLM reaches 45.78% on EmoSum versus 42.52% on IEMOCAP, suggesting the synthetic data captures realistic emotional cues.

• Emotion labels are relatively balanced. 45.72% of utterances are Neutral, 32.3% Happiness, 9.07% Sadness, 4.76% Surprise, 2.99% Fear, 2.32% Other, 1.68% Anger, and 1.16% Disgust.

• Human raters place the corpus second overall. Twelve raters scored 480 audio segments (20–30 seconds each) on a 1–5 scale. Spoken DialogSum averages 3.78 (Naturalness 3.64, Emotion Expressivity 3.84, Emotion Consistency 3.75, Sound Quality 3.89), behind MELD at 4.12 (4.06 / 4.46 / 4.36 / 3.58) but ahead of DailyTalk (3.52) and Behavior-SD (3.31). Its sound quality of 3.89 exceeds Switchboard's 2.88 and MELD's 3.58.

• Scale far exceeds comparable emotion-annotated corpora. Spoken DialogSum offers roughly 160 hours against MELD's 12 hours and DailyTalk's 20 hours. Note that the paper states 13,460 dialogues in the abstract, body, and Table 4, but lists 13,640 in Table 5(a); duration is given as 159.87 hours in Table 4 and as approximately 165 hours in the introduction and conclusion.

Methodology in Plain English

The authors started from DialogSum, a text dataset of scripted daily-life dialogues with human-written summaries, and treated it as raw material to be converted into realistic speech.

First, LLAMA3.3 70B rewrote each scripted dialogue using real Switchboard transcripts as a style guide, adding fillers and hesitations while preserving every speaker, line order, and meaning. A second pass inserted back-channels — brief listener acknowledgements such as short reactions — using a special {X: backchannel} marker and Switchboard examples to keep the wording varied. Back-channels were deliberately designed to overlap with the other speaker's speech, increasing difficulty. GPT-4o-mini then evaluated the rewritten dialogues to avoid the self-preference bias that occurs when the generating LLM judges its own output.

Next, the same GPT-4o-mini was prompted with the full dialogue and asked to assign each utterance one of eight emotions (Happiness, Sadness, Disgust, Fear, Surprise, Anger, Other, Neutral) as an 8-dimensional one-hot vector, plus discretized pitch and speaking-rate values on a 0/1/2 scale.

For speech, the authors built a speaker bank by annotating GigaSpeech recordings for age, gender, pitch, expressiveness, and speaking rate, using a pretrained Wav2Vec2 age/gender estimator and the PENN pitch tracker. They retained 20,385 voices, filtering to prompts longer than 5 seconds from audiobook or podcast sources and excluding excessively monotone recordings. Zonos-hybrid, a conditional TTS model whose backbone interleaves Mamba-style state-space blocks with Transformer layers, then synthesized the audio conditioned on the emotion vector and prosody values. To counteract under-expressive output on short utterances, the authors raised the baseline pitch and speaking-rate inputs (mapping low/medium/high pitch to 60.0/85.0/110.0 and speaking rate to 15.0/18.0/21.0 phonemes per second) and forced all back-channels to the slowest rate with one second of appended silence.

Finally, overlapping turns were placed using timing statistics from CANDOR: interruptions were positioned roughly 1.5 seconds before the end of the host's turn (building on CANDOR's normal distribution of 0.45 s ± 0.05 s before the previous speaker finishes, plus a 1-second buffer), and back-channels were placed at the start of the following turn, consistent with CANDOR's 0.2 s ± 0.02 s delay.

Evaluation used a test split of 500 dialogues, each with three human-written summaries, scored by averaging across references. Baselines included transcript-only LLaMA-2-7B-chat, a cascaded Whisper Large V2 + LLaMA-2-7B-chat pipeline, and five end-to-end audio models: WavLLM, Qwen-Audio-Chat, Audio-Flamingo3, LTU-AS, and SALMONN (7B and 13B).

Why This Matters

Impact on research. Spoken dialogue summarization has been constrained by the absence of data connecting audio, summaries, and paralinguistic cues. Spoken DialogSum gives researchers a controllable testbed for measuring whether models genuinely integrate acoustic affect with lexical meaning, or merely transcribe and summarize. The three-task design separates semantic abstraction from paralinguistic perception from cross-modal fusion, making it possible to diagnose where a system fails.

Real-world applications:

• Meeting and call-center summarization, where a summary needs to capture not just decisions but the emotional tenor of the exchange.
• Assistive and accessibility tools that summarize spoken conversations for deaf and hard-of-hearing users, where affect is often as important as content.
• Mental health and counseling-adjacent research and training tools that need to track speaker emotion over time.
• Conversational agent evaluation, where the corpus provides emotion-labeled multi-speaker audio for testing whether dialogue systems respond appropriately to user affect.

Industry relevance. The finding that end-to-end Audio-LLMs substantially outperform cascaded ASR-then-LLM pipelines on emotion-rich summarization suggests that products built on speech-to-text followed by a text LLM may be discarding acoustic information that matters. The corpus also offers training and evaluation data for voice assistants, customer-service analytics, and any system that has to reason about how something was said, not just what was said, including the overlap and interruption patterns that make real conversation hard to process.

Future Directions

• Public dataset release. Only an online demo page is currently available; the authors state that the full dataset will be released in the near future, which is a prerequisite for reproducible benchmarking by other groups.
• Closing the synthetic-to-real gap. The corpus is TTS-generated. Whether models trained or evaluated on it transfer to spontaneous human speech, and how the human evaluation gap versus MELD (3.78 average versus 4.12) affects that transfer, remains open.
• Improving paralinguistic modeling in Audio-LLMs. WavLLM and Qwen-Audio lag well behind a dedicated wav2vec 2.0 classifier on age and gender (for example, 31.4% versus 66.3% age accuracy for WavLLM), indicating that general-purpose audio LLMs do not yet capture fine-grained speaker traits without explicit supervision.
• Reconciling reported figures and expanding scope. The dialogue count (13,460 versus 13,640), total duration (159.87 versus approximately 165 hours), and the relative ROUGE-L gain (28% versus 29%) are reported inconsistently across sections, and extension to additional languages, domains, and non-synthetic recordings are natural next steps.

Target Audience

This paper is most useful to researchers and engineers working on spoken language processing, multimodal dialogue systems, and speech-language model evaluation — particularly those building or benchmarking Audio-LLMs, TTS pipelines for conversational data, or emotion-aware summarization systems. Dataset builders interested in synthetic-to-real data generation will find the three-stage pipeline and its prompt templates directly applicable. Practitioners in call-center analytics, accessibility, and conversational AI product development can use the benchmark results to decide between cascaded and end-to-end architectures. The paper assumes some familiarity with ASR, TTS, and ROUGE-style evaluation, placing it at an intermediate level.

Authors’ abstract

Recent audio language models can follow long conversations. However, research on emotion-aware or spoken dialogue summarization is constrained by the lack of data that links speech, summaries, and paralinguistic cues. We introduce Spoken DialogSum, the first corpus aligning raw conversational audio with factual summaries, emotion-rich summaries, and utterance-level labels for speaker age, gender, and emotion. The dataset is built in two stages: first, an LLM rewrites DialogSum scripts with Switchboard-style fillers and back-channels, then tags each utterance with emotion, pitch, and speaking rate. Second, an expressive TTS engine synthesizes speech from the tagged scripts, aligned with paralinguistic labels. Spoken DialogSum comprises 13,460 emotion-diverse dialogues, each paired with both a factual and an emotion-focused summary. We release an online demo at https://fatfat-emosum.github.io/EmoDialog-Sum-Audio-Samples/, with plans to release the full dataset in the near future. Baselines show that an Audio-LLM raises emotional-summary ROUGE-L by 28% relative to a cascaded ASR-LLM system, confirming the value of end-to-end speech modeling.