VowelPrompt: Hearing Speech Emotions from Text via Vowel-level Prosodic Augmentation

AI summary

Overview

• Research area: Natural Language Processing, specifically speech emotion recognition (SER) with large language models, multimodal/paralinguistic understanding, and interpretable prompting.
• Technical level: Intermediate — assumes familiarity with LLM fine-tuning, reinforcement learning from verifiable rewards (RLVR), forced alignment, and standard paralinguistic features (pitch/F0, intensity, duration).
• Scope: The paper introduces a method that converts vowel-level acoustic prosody into natural-language descriptions appended to transcripts, then adapts LLMs to it via supervised fine-tuning and GRPO-based reinforcement learning, evaluated on five emotion recognition benchmarks across four languages.

What This Paper Is About

Emotion in speech depends on both what is said (words) and how it is said (pitch, loudness, timing), but text-only LLMs only see the words, so they miss fine-grained vocal expressivity. Existing prompt-based fixes describe prosody at the coarse sentence level, which blurs local detail. This paper asks whether converting the prosody of individual vowel segments — the parts of speech that carry most of the energy and intonation — into plain-language labels can make an LLM noticeably better at emotion recognition while remaining interpretable and requiring no audio at inference time.

Key Contributions

1. Vowel-level prosodic descriptors. VowelPrompt uses forced alignment to locate vowel segments, extracts low-level descriptors (pitch level, pitch slope, pitch variation, intensity level, intensity variation, duration), applies speaker-level and vowel-type normalization plus quantile binning, and maps the values deterministically into human-readable phrases such as "high F0, rising, loud, lengthened." These are appended to the transcript so the LLM reasons jointly over lexical semantics and localized prosody — unlike opaque acoustic embeddings.

2. A two-stage adaptation pipeline. Supervised fine-tuning (SFT) on a small portion of data with gold reasoning traces from a high-capacity text-only LLM (e.g., GPT-4o) provides cold-start alignment, followed by Reinforcement Learning with Verifiable Reward (RLVR) via Group Relative Policy Optimization (GRPO) to improve structured output adherence, reasoning quality, and robustness, with a KL penalty constraining drift from the SFT reference model.

3. Multilingual extension via IPA. Vowel inventories are unified across languages using the International Phonetic Alphabet, with Montreal Forced Aligner (MFA) for phoneme-level alignment (supporting over 20 languages) and language-level z-score normalization, enabling evaluation on English, German, French, and mixed-lingual data.

4. Broad empirical validation. Experiments on five benchmarks (IEMOCAP, MELD, CaFE, EmoDB, ASVP-ESD) across zero-shot, few-shot, fine-tuned, cross-domain, and cross-linguistic conditions, plus a large appendix of ablations, counterfactual tests, human evaluation of reasoning traces, and comparisons against self-supervised acoustic baselines.

Main Findings

• Zero-shot gains over both baselines. With GPT-4o, VowelPrompt improves over the transcript-only Zero-Shot Baseline by up to 7.80% UACC and 7.11% WF1 on IEMOCAP and up to 2.19% UACC and 3.25% WF1 on MELD, and it beats SpeechCueLLM (sentence-level prosody descriptions) in every setting. On IEMOCAP with Transcript & Context and GPT-4o, VowelPrompt scores 62.26 UACC / 60.74 WF1, versus 60.07 / 58.52 for SpeechCueLLM and 55.51 / 53.63 for the baseline.

• The gains transfer to a smaller model. With LLaMA-3-8B-Instruct, VowelPrompt yields up to 3.64% UACC and 3.63% WF1 improvements over the baselines, indicating a model-agnostic, portable prompting strategy that needs no task-specific fine-tuning.

• Fine-tuning superiority on Weighted F1. Under SFT with LLaMA-3-8B-Instruct, VowelPrompt reaches 73.46 on IEMOCAP and 69.61 on MELD, with absolute improvements of up to 3.14% and 2.17% respectively; with LLaMA-4-Scout-17B-16E-Instruct it reaches 73.85 / 70.12. After GRPO refinement it reaches 74.02 (IEMOCAP) and 69.79 (MELD) on the larger model, outperforming sentence-level prosody descriptions by as much as 1.47% and 1.88%.

• Stronger cross-domain generalization. Transferring from IEMOCAP to MELD under SFT & GRPO, VowelPrompt scores 60.28, a 5.12% improvement over SpeechCueLLM; transferring from MELD to IEMOCAP gives 51.75, a 6.96% improvement. Gains are modest in the zero-shot condition and grow substantially with supervised adaptation.

• Working multilingual performance. Zero-shot with GPT-4o, VowelPrompt gets 51.42 WF1 on French CaFE (vs. 45.10 transcript-only) and 69.85 on German EmoDB (vs. 64.86 transcript-only). On mixed-lingual ASVP-ESD with Qwen2-7B-Instruct, it reaches 70.54 under SFT and 71.36 under SFT & GRPO.

• No single feature dominates. Removing any one of the six vowel-level descriptors under SFT with LLaMA-3-8B-Instruct keeps scores above 72.5% on IEMOCAP and 69.05% on MELD, but every removal hurts slightly; pitch-related features (level, slope, variation) show the most noticeable impact.

• Few-shot also improves. Adding three labeled in-context exemplars yields consistent WF1 gains of roughly 0.8% to 1.2% over zero-shot across methods, with VowelPrompt remaining best; for example, GPT-4o with Transcript & Context goes from 53.63 to 54.42 on IEMOCAP and from 63.57 to 64.51 on MELD.

• Dataset scope. Five benchmarks are used: IEMOCAP (English, 5 emotions, 10 speakers, 5531 utterances, 7.0 hours), MELD (English, 7 emotions, 304 speakers, 13706 utterances, 12.1 hours), CaFE (French, 7 emotions, 12 speakers, 936 utterances, 1.2 hours), EmoDB (German, 7 emotions, 10 speakers, 535 utterances, 0.5 hours), and ASVP-ESD (mixed language, 12 emotions, 131 speakers, 13964 utterances, 18.0 hours).

Methodology in Plain English

1. Find the vowels. Each utterance comes with a transcript. A phoneme-level forced-alignment tool (MFA) marks exactly when each sound starts and ends; segments matching a vowel inventory derived from the IPA are kept, consonants are discarded.

2. Measure the prosody inside each vowel. For every vowel segment the authors compute six numbers: mean pitch (F0), pitch slope (rising or falling), pitch variation (standard deviation of F0), mean intensity (RMS energy), intensity variation, and duration in seconds. Pitch and intensity use Praat-style signal processing with speaker-adaptive floor and ceiling settings.

3. Make the numbers comparable and human-readable. Values are z-normalized per speaker and then per vowel type, then discretized into K ordinal bins (e.g., "very low" through "very high"). A fixed, parameter-free mapping turns these bins into short phrases like "high F0, rising, loud, lengthened," which are appended to the transcript. Because the mapping is deterministic and the output is text, a text-only LLM can use it with no audio encoder at inference.

4. Adapt the LLM in two stages. First, SFT on a small slice of training data using gold reasoning traces generated by a high-capacity text-only LLM, teaching the model both the label and an explanation. Second, RLVR with GRPO: the model emits a reasoning trace inside <think> tags and a final label inside <answer> tags, and receives a composite reward — 1 for a correct label, 0 otherwise, plus 1 for valid tag structure, 0 otherwise. Both reward terms are deterministic and parameter-free, so no learned reward model is needed. A KL penalty keeps updates close to the SFT model. Fine-tuning uses LoRA-based parameter-efficient tuning on 20% of training data.

5. Extend across languages. Aligned phonemes from each language are mapped onto a shared IPA inventory, and prosodic features get an additional language-level z-normalization for English, German, and French, after which the same prompt template and training objectives are reused.

Why This Matters

Impact on research. The work bridges two traditions that are usually separate: interpretable phonetic analysis (openSMILE/GeMAPS-style low-level descriptors) and modern LLM reasoning. It shows that a discretized, human-readable intermediate representation can rival or beat black-box acoustic embeddings for emotion recognition while remaining inspectable — and it provides a template for using RLVR with verifiable, rule-based rewards in a paralinguistic task. The extensive appendix (counterfactual transcript shuffles, prosody permutations, placebo controls, vowel-alignment perturbation, consonant comparisons, human evaluation of reasoning traces) sets a bar for validating that a model is actually using the cues it claims to use.

Real-world applications.

• Customer-service and call-center analytics: flagging frustration or agitation in support calls without storing or processing raw audio at inference time.
• Mental health and wellbeing monitoring: tracking vocal affect shifts over time in telehealth or conversational agents, with explanations a clinician can read.
• Human-computer interaction and voice assistants: adapting response tone to a user's emotional state in real time.
• Content moderation and media analysis: detecting emotional intensity in dubbed, subtitled, or transcribed multilingual media where only text pipelines are available.

Industry relevance. Because inference requires only text, VowelPrompt can be layered onto existing ASR-plus-LLM pipelines without an audio encoder in the serving path, which lowers latency and infrastructure cost. Its model-agnostic behavior across GPT-4o, LLaMA-3-8B-Instruct, LLaMA-4-Scout-17B-16E-Instruct, and Qwen2-7B-Instruct, and its cross-domain robustness, make it attractive for production systems where data distributions shift between training and deployment.

Future Directions

• Which features and how many bins are optimal. The paper reports ablation results for removing individual descriptors and mentions a study of the number of quantization bins K, but the specific optimal K and full bin-count results are not reported in the provided content; systematically tuning descriptor sets and bin granularity remains open.
• Beyond discrete emotion categories. The introduction notes that SER can also be framed in the valence–arousal–dominance space, but all reported experiments use discrete labels; extending vowel-level prompting to dimensional affect is a natural next step.
• Robustness limits of alignment and speech rate. The authors include studies on perturbed vowel boundaries and on speech-rate regimes, but quantitative outcomes are not reported here; knowing how much alignment error the method tolerates before degrading is important for low-resource or noisy deployment.
• Broader language and modality coverage. IPA-based mapping is demonstrated on English, German, and French plus mixed-lingual data; scaling to many more languages, and to settings that combine vowel and consonant cues, are logical extensions the authors themselves frame as comparisons.

Target Audience

Researchers and practitioners working on speech emotion recognition, paralinguistics, and multimodal or speech-aware LLMs; engineers building interpretable, text-only affect pipelines on top of ASR; and phoneticians or speech scientists interested in how segment-level prosodic evidence can be operationalized inside modern language models. Readers with a basic grounding in LLM fine-tuning and speech features will get the most out of it, while newcomers can follow the core idea — label each vowel's prosody in words, then teach the model to reason over those words.

Authors’ abstract

Emotion recognition in speech presents a complex multimodal challenge, requiring comprehension of both linguistic content and vocal expressivity, particularly prosodic features such as fundamental frequency, intensity, and temporal dynamics. Although large language models (LLMs) have shown promise in reasoning over textual transcriptions for emotion recognition, they typically neglect fine-grained prosodic information, limiting their effectiveness and interpretability. In this work, we propose VowelPrompt, a linguistically grounded framework that augments LLM-based emotion recognition with interpretable, fine-grained vowel-level prosodic cues. Drawing on phonetic evidence that vowels serve as primary carriers of affective prosody, VowelPrompt extracts pitch-, energy-, and duration-based descriptors from time-aligned vowel segments, and converts these features into natural language descriptions for better interpretability. Such a design enables LLMs to jointly reason over lexical semantics and fine-grained prosodic variation. Moreover, we adopt a two-stage adaptation procedure comprising supervised fine-tuning (SFT) followed by Reinforcement Learning with Verifiable Reward (RLVR), implemented via Group Relative Policy Optimization (GRPO), to enhance reasoning capability, enforce structured output adherence, and improve generalization across domains and speaker variations. Extensive evaluations across diverse benchmark datasets demonstrate that VowelPrompt consistently outperforms state-of-the-art emotion recognition methods under zero-shot, fine-tuned, cross-domain, and cross-linguistic conditions, while enabling the generation of interpretable explanations that are jointly grounded in contextual semantics and fine-grained prosodic structure.