SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis

AI summary

Overview

Research area: Speech sentiment analysis, speech emotion recognition, paralinguistics, multimodal and text-only large language models, synthetic speech data.

Technical level: Intermediate. The paper is readable without deep signal-processing background, but it assumes familiarity with terms such as prosody, F0, automatic speech recognition (ASR), word error rate (WER), and parameter-efficient fine-tuning. The dataset itself is described in largely non-mathematical terms.

Scope in one sentence: The paper introduces SpeechSense, an 8-class, paralinguistics-focused synthetic speech dataset with human validation, and shows experimentally that models with acoustic access outperform text-only models at detecting fine-grained interpersonal stances.

What This Paper Is About

Speech sentiment analysis systems often work by transcribing speech to text with ASR and then running text sentiment analysis, which throws away prosody, tone, pauses, stress, and tempo — the very cues that signal how something is said rather than what is said. Existing benchmarks also tend to label broad emotions such as happy or sad rather than nuanced interpersonal stances such as confident or impatient. The paper's goal is to build SpeechSense, a dataset and label taxonomy that isolates the acoustic "how" from the lexical "what," so that fine-grained speaker attitudes can be detected mainly from prosody.

Key Contributions

1. A fine-grained, paralinguistics-focused label set. The authors define a specialized 8-class taxonomy — Confident, Nervous, Warm, Apathetic, Passionate, Impatient, Sarcastic, and Neutral — organized into four comparative attribute groups (Internal Certainty, High-Energy Valence, Social Connection, Prosodic Deviation), with each label given an acoustic and psychological definition.

2. A curated dataset combining high-fidelity synthesized speech with human validation. SpeechSense is built through a three-stage pipeline: semantic-prosodic decoupled carrier text generation with Qwen3-Max, role-play synthesis using Lovo.ai, and dual-stage human validation and filtering via Qualtrics and Prolific. The final curated test set contains 669 clips, with an overall Fleiss' Kappa of 0.4437.

3. An architecture-spanning validation. The paper benchmarks multi-modal LLMs (Qwen2.5-Omni-3B and 7B), text-only LLMs (Qwen2.5-Instruct-3B and 7B), and speech encoders (Whisper-large-v3, HuBERT-large, Wav2Vec2-large) under a unified frozen-backbone-plus-linear-head protocol to test whether acoustic access is necessary.

4. Public release under CC BY 4.0. Audio, labels, and synthesis directives are released along with supplementary analyses at the project's GitHub repository.

Main Findings

• Zero-shot performance is near random. Across all model families, zero-shot Macro-F1 ranges from 1.31% to 6.63%, indicating that pre-trained multi-modal LLMs and speech encoders do not inherently possess fine-grained attitudinal representations.

• Acoustic access drives large gains after training. Multi-modal LLM audio models improve by up to 50 percentage points in accuracy. Qwen2.5-Omni-7B in audio mode reaches 56.95% accuracy and 56.76% Macro-F1, and Qwen2.5-Omni-3B in audio mode reaches 54.86% accuracy and 53.38% Macro-F1.

• Text-only models fail and collapse inconsistently. After supervised training, Qwen2.5-Omni text models saturate at 20–22% F1, while Qwen2.5-Instruct-7B reaches only 15.97% F1 and the Instruct-3B variant drops to 4.60% F1 — lower than its own zero-shot baseline of 6.63%. The four text-only configurations each collapse toward a different dominant class (Omni-3B concentrates nearly 70% of predictions on Impatient; Omni-7B favors Sarcastic; Instruct-3B directs 87% to Warm; Instruct-7B spreads across Confident, Sarcastic, and Impatient), and Neutral is never predicted by any of the four.

• The best audio model substantially beats the best text model. The gap between 56.95% accuracy (best audio) and 26.76% accuracy (best text) supports the claim that the task is solvable almost exclusively from paralinguistic rather than linguistic information.

• Speech encoders capture real attitudinal signal without language modeling. Whisper-large-v3 (45.06% Macro-F1), HuBERT-large (43.79%), and Wav2Vec2-large (42.45%) converge within a narrow 2.6-point band despite distinct pre-training paradigms. Multi-modal LLM audio models (53–57% F1) still exceed them by roughly 10–14 percentage points, suggesting the language component adds reasoning capacity on top of prosodic features.

• Nervous is the most reliably detected attitude in acoustic models (68–80% F1), while Neutral and Confident are the hardest (below 45% F1 for most models), consistent with attitudes defined by the absence of distinctive prosodic excursions. Errors concentrate on prosodically adjacent pairs: Confident–Neutral mutually, Confident–Passionate in multi-modal LLMs, and Warm–Sarcastic in speech encoders.

• Scaling helps selectively. For multi-modal LLM audio models, moving from 3B to 7B raises Macro-F1 from 53.38% to 56.76%, with the gain concentrated on cognitively complex classes such as Sarcastic and Confident.

• Synthesis quality is high. Measured with Whisper-large-v3, the benchmark achieves an average word error rate of 3.70% on the curated test set and 5.42% on the training set. Sarcastic shows the highest WER in both (7.98% test, 8.57% train).

• Human agreement is moderate and comparable to established corpora. Fleiss' Kappa of 0.4437 for SpeechSense compares with CREMA-D (κ = 0.42) and IEMOCAP (κ = 0.40), and substantially exceeds EmoNet-Voice's Krippendorff's α = 0.14.

Methodology in Plain English

The authors wanted audio where the meaning comes from delivery, not words, so a model cannot cheat by reading the transcript. They did this in three stages.

Stage 1 — neutral carrier text. Using Qwen3-Max, they generated 120 carrier sentences per label. The sentences deliberately exclude explicit emotional words, are semantically neutral so they can be reused across labels, vary in structure (declarative, imperative, conditional) to prevent syntactic overfitting, and run 3–8 seconds. Because the text is neutral, any sentiment must come from the voice.

Stage 2 — role-play synthesis. Authentic recordings of these fine-grained attitudes are scarce, privacy-sensitive, and ethically constrained, so the authors synthesized audio instead. After evaluating six TTS systems — open-source (CosyVoice, IndexTTS2, Kokoro) and proprietary (GPT-4o, ElevenLabs) among them — they selected Lovo.ai for its human-like prosody, speaker diversity (30+ voice profiles), and scalable API. Rather than twisting pitch knobs, they gave the engine situational acting directives grounded in the Stanislavski method, for example configuring Sarcastic as "reading like a victim of a prank with dry thanks." Holding the text constant across attitudes isolates prosody as the only differentiator.

Stage 3 — human validation and filtering. Clips were checked by annotators recruited on Prolific (eligible pool 23,006 from 150,000+; native English; Bachelor's or higher; approval rate ≥99%) and administered through Qualtrics, with at least three annotators per clip. Of 960 pre-filtering clips, 623 (93.12% of the final set) were retained by majority vote (3/3 or 2/3 agreement) and 46 (6.88%) by a Reference Alignment rule, which keeps ambiguous clips if at least one annotator matches the intended synthesis target. The result is 669 curated test clips (69.7% retention).

Evaluation setup. The training set is larger and separate — 1,522 clips generated with a different LLM family (Gemini 3 Pro) under weak supervision — so training and testing use different text sources and the model cannot fit one LLM's writing style. Both subsets use the same 30 voice profiles, but speaker–label balance is controlled: 26 of 30 speakers cover all 8 attitudes and the remaining 4 cover 7, so speaker identity offers no shortcut. All models use the same protocol: frozen backbone plus a linear classification head with cross-entropy loss. LLMs get LoRA on attention modules; speech encoders are trained in two stages (head only, then full fine-tuning).

Why This Matters

Impact on research. The paper reframes speech sentiment analysis away from basic emotion categories and text-centric ASR-then-classify pipelines, and provides a controlled testbed where text is held constant so that measured performance reflects acoustic understanding. The near-random zero-shot scores and the failure of text-only models under training give a concrete empirical argument that fine-grained stance detection requires acoustic signal, and the reported inter-annotator agreement positions the dataset alongside established corpora such as CREMA-D and IEMOCAP.

Real-world applications:

• Online recruitment and job interviews, where distinguishing a confident from a nervous delivery carries information that the transcript alone misses.
• Customer service, where detecting attitudes such as impatient or warm in a caller's voice can guide routing and escalation.
• Healthcare, where paralinguistic cues in patient speech can be relevant to assessment.
• Voice assistants and spoken dialogue systems, which can adapt tone and responses based on the speaker's stance rather than just their words.

Industry relevance. The cascade paradigm — ASR followed by text sentiment analysis — dominates deployed systems, and the paper shows it is structurally limited for attitudinal tasks: ASR error propagation degrades downstream sentiment detection, and converting audio to text irreversibly strips paralinguistics. That makes the case for end-to-end audio processing, speech encoders, and multi-modal LLMs in products where reading the speaker's attitude matters. The open release under CC BY 4.0, including synthesis directives, lowers the barrier for teams that cannot collect sensitive attitudinal speech themselves.

Future Directions

• Closing the synthetic-to-real gap. SpeechSense is built entirely from synthesized speech. The authors position it as a foundational cold-start resource and call for future work on real-world adaptation, since a domain gap between synthetic and real speech may exist.
• Expanding the taxonomy and languages. The current label set covers eight interpersonal stances in English only; extending to additional attitudes and languages would broaden applicability.
• Increasing speaker diversity. Speaker diversity is limited to 30 voice profiles from a single TTS engine. Future iterations could incorporate multiple engines and a larger speaker pool to improve generalization, even though the current speaker–label balance analysis shows no identity leakage.
• Improving the hardest classes and adjacent-pair confusions. Neutral and Confident remain below 45% F1 for most acoustic models, and systematic confusions persist between Confident–Neutral, Confident–Passionate, and Warm–Sarcastic, leaving open the question of how to separate prosodically adjacent stances.

Target Audience

Researchers and practitioners working on speech emotion recognition, speech sentiment analysis, paralinguistics, and spoken dialogue systems; developers building multi-modal or audio-native LLM applications who need a benchmark for attitudinal understanding; dataset builders interested in synthetic data generation with human validation; and applied teams in recruitment, customer service, and healthcare-facing speech products who want evidence on whether text-only pipelines suffice for fine-grained speaker attitudes.

Authors’ abstract

Recent advances in AI have revolutionized speech processing, yet effective speech understanding requires discerning not just what is said, but how it is said. Speech Sentiment Analysis plays a critical role in decoding these paralinguistic cues for diverse real-world applications such as recruitment and customer service. However, existing Speech Sentiment Analysis research faces two primary limitations. First, dominant approaches rely on text-centric pipelines that cascade Automatic Speech Recognition with text analysis. This process inevitably discards essential acoustic features like prosody and tone, failing to capture attitudinal meanings in acoustically ambiguous utterances. Second, current benchmarks suffer from a mismatch in label granularity, prioritizing basic emotions (e.g., happy, sad) over the nuanced interpersonal stances (e.g., confident, impatient) necessary for social sensitivity. To address these limitations, we propose a novel dataset, SpeechSense, for fine-grained speech sentiment analysis. Specifically, we define a specialized 8-class taxonomy of interpersonal stances detectable primarily through prosodic cues beyond lexical content alone. We then construct a curated dataset based on this taxonomy, built from high-fidelity speech synthesis and rigorous human validation. Comprehensive experiments across multi-modal LLMs, text-only LLMs, and speech encoders demonstrate that models with acoustic access consistently outperform text-only baselines. These results empirically validate the primacy of acoustic cues in detecting subtle speaker attitudes, highlighting the necessity of SpeechSense. Dataset and supplementary materials are available at https://github.com/Sher13cked/SpeechSense.