Style Amnesia: Investigating Speaking Style Degradation and Mitigation in Multi-Turn Spoken Language Models

AI summary

Overview

Research area: Spoken language models (SLMs) and paralinguistic style control in multi-turn spoken dialogue — combining speech generation, instruction following, and evaluation methodology for expressive speech.

Technical level: Intermediate. Readers should be comfortable with concepts such as instruction-following rates, attention weights, cascaded speech pipelines (ASR + LLM + TTS), and automatic judges for speech attributes.

Scope (one sentence): The paper identifies, quantifies, and partially mitigates "style amnesia" — the tendency of spoken language models to abandon a speaking style instruction given at the start of a conversation as the dialogue progresses — across five SLMs, four paralinguistic attributes, and 100 dialogue topics.

What This Paper Is About

Users of spoken language models may want the model to keep speaking in a chosen style — sad, fast, quiet, Indian-accented — for an entire conversation, and it is unrealistic to expect them to repeat that instruction every turn. This paper asks whether SLMs can hold a style set once at the beginning of a multi-turn spoken dialogue, and finds that they cannot: style instruction-following drops sharply after the first turn. The authors name this failure style amnesia, measure it systematically, diagnose why it happens, and show a prompting-based workaround.

Key Contributions

1. Identification of style amnesia. The authors show that in multi-turn spoken conversations, SLMs fail to keep following a speaking style instruction issued in the first turn, and that the instruction-following rate typically peaks at turn one and declines afterwards.
2. A recall-based mitigation. They demonstrate that explicitly prompting the SLM to restate the initial style instruction before each user turn reduces the degradation rate, even though the same models already retain the instruction.
3. Diagnostic analysis of the mechanism. They track attention weights to the style instruction tokens, compare system-message versus user-message placement of the instruction, and examine text–acoustic synergy for emotion and speaking-rate styles.
4. A reusable evaluation framework with validated judges. They build a user-simulator-driven, turn-level evaluation over 10 speaking styles and 100 topics (1,000 dialogues per model) and validate the emotion and accent judges against human annotators.

Main Findings

• Style amnesia is widespread. None of the three proprietary SLMs (GPT-4o, GPT-4o mini, Gemini Live) or two open-source SLMs (Qwen2.5-Omni, Step-Audio 2 mini) maintained a consistent speaking style when instructed to do so at the start of a conversation.
• The cascaded baseline is stable. A cascaded system of GPT-5 mini plus Gemini-TTS, in which the TTS receives the style instruction every turn and which serves as the performance upper bound, showed degradation within 3.0%.
• Emotion degradation is large for some models. Gemini Live and GPT-4o showed degradation rates ranging from 13.7% to 26.7% for Anger and Sadness. GPT-4o mini reached 34.7% for Anger and 65.3% for Sadness.
• Low degradation can reflect weak first-turn control. Qwen2.5-Omni and Step-Audio 2 mini showed around 14.0% degradation for Sadness, but only 3.7% and 1.0% for Anger, because both already struggled to produce Anger in the first turn.
• Accent results vary by model. Gemini Live was stable at maintaining an Indian English accent; GPT-4o mini showed nearly 50% degradation for it; Step-Audio 2 mini had a first-turn instruction-following rate of 32.0% with a 5.0% degradation rate; Qwen2.5-Omni performed well on the North American accent but failed to produce the Indian accent.
• Models favor their default styles. Nearly all SLMs performed better on happiness, neutral tone, and North American English, which the authors hypothesize are the default speaking styles of the evaluated models; the emotion and accent distributions of samples generated during speed and volume evaluations support this.
• Loud is harder than quiet. For the Volume style, speaking loudly was generally more challenging than speaking quietly, even in the first turn, though volume-capable models still degraded over turns.
• Speed control degrades. Most SLMs controlled speed reasonably in the first turn but degraded significantly; Qwen2.5-Omni failed even in the first turn, with a first-turn instruction-following rate below 50%, comparable to a random baseline in the pairwise comparison setting.
• Attention to the style instruction collapses. For Step-Audio 2 mini under speed instructions, the average attention weight on style instruction tokens was 8.55% (Slow) and 8.30% (Fast) in turn 1, falling to 0.59% and 0.56% by turn 4, with 1.70%/1.51% at turn 2 and 0.90%/0.85% at turn 3.
• System messages are worse than user messages for style. When the style instruction was placed in a system message, GPT-4o, GPT-4o mini, and Step-Audio 2 mini showed drops of approximately 30%, 50%, and 20% for sadness; GPT-4o mini showed nearly an 80% drop for the Indian English accent; and speed instructions were nearly ignored, with performance comparable to the random baseline. Style amnesia occurred in both placements.
• The models remember but do not comply. Proprietary models showed near-perfect recall of the initial instruction across turns, and Step-Audio 2 mini showed recall rates from 55.0% to 89.0%. GPT-4o mini, despite 65.3% degradation for Sadness and 20% for speaking slowly, still recalled the instruction well — so the failure is not memory loss.
• Recall reduces degradation. Adding the recall process reduced degradation: GPT-4o mini achieved roughly a 25% reduction in average degradation rate (for example, Indian accent from 49.7 to 14.9, Sadness from 65.3 to 30.3). Step-Audio 2 mini improved only slightly or not at all on some tasks, which the authors attribute to its lower recall rate.
• Semantic and acoustic emotion degrade together. For emotion, both the textual (GPT-5 mini classifier on the transcription) and acoustic (Emotion2vec-Large) measures of style adherence suffered from style amnesia simultaneously.
• Text and acoustic strategies for speed differ. Gemini Live produced fewer words under the fast condition, leveraging conciseness; GPT-4o and Step-Audio 2 mini generated comparable or more words while compressing them into shorter durations. The words-per-minute gap between fast and slow conditions narrowed over turns for all models.
• Judges are validated. Against human annotation on 720 speech clips, Voxlect achieved Cohen's Kappa 0.809 and MCC 0.811 for accent, beating Gemini-2.5 Pro (0.741 / 0.747). Emotion2vec-Large achieved Cohen's Kappa 0.476 and MCC 0.511 for emotion, slightly above Gemini-2.5 Pro (0.464 / 0.487).
• Judges agree on semantic coherence. All evaluated SLMs maintained reasonable semantic consistency across turns.
• Pitch also shows amnesia. In supplementary experiments, Gemini Live's Low Pitch instruction-following rate fell from 67.0% at turn 1 to 54.0% at turn 3, and GPT-4o's High Pitch rate fell from 89.0% at turn 1 to 75.6% at turn 4; Step-Audio 2 mini stayed close to the 50% random baseline.
• Composite styles also degrade. With instruction A in the first user turn and instruction B in the second, GPT-4o's joint instruction-following rate for a composite condition dropped from 88.0% at Turn 2 to 59.3% at Turn 4.

Methodology in Plain English

The authors set up each evaluation as a spoken conversation between the model under test and a simulated user. At the start of the dialogue, the model is told to speak in one specific paralinguistic style for the whole conversation — one of four emotion values (sad, happy, angry, neutral), two accents (North American, Indian), two volume levels (higher, lower), or two speeds (faster, slower). That gives 10 styles, and each is paired with 100 conversation openers drawn from the Soda dialogue dataset (topics were generated with GPT-5 mini and manually filtered), producing 1,000 dialogues per model. Conversations run for four assistant turns (K = 4).

The "user" is a cascaded system: an ASR module transcribes the model's speech (skipped when the model already outputs text), GPT-5 mini generates the next user reply, and GPT-4o mini TTS speaks it back. The user simulator is instructed to keep each response under 20 English words and to avoid phrasing that cannot be spoken aloud. Models were run at temperature 1, since greedy decoding caused long silences at the end of audio for some models.

Each generated response is judged against the target style. Emotion is judged by Emotion2vec-Large (focusing on happiness, sadness, anger, and neutral), accent by Voxlect-English-Dialect-Whisper-Large-v3 (focusing on North American and Indian English), and volume and speed by deterministic comparisons against a neutral-baseline rendering from the same model — LUFS via PyLoudnorm for volume, words per minute via Parakeet TDT v2 for speed. Because "loud" and "fast" have no absolute definition, the judges test whether the styled output is louder/faster (or quieter/slower) than the neutral baseline for the same content.

Two headline metrics are reported: the first-turn instruction-following rate, which shows how well the model handles the style before any conversation buildup, and the degradation rate, the average drop in instruction-following across later turns relative to the first, counting only decreases. In a second experiment, after the first turn the model is asked to restate the original style instruction before answering; a recall rate measures whether the restatement matches, and the degradation rate is recomputed with the recall process in place. For mechanistic insight, the authors inspect attention weights over the style instruction tokens inside Step-Audio 2 mini, and they repeat the style setup with the instruction placed in a system message instead of the first user message.

Why This Matters

Impact on research. The paper adds a turn-level, style-focused dimension to SLM evaluation that prior benchmarks (SpokenWOZ, C³, ContextDialog, Vstyle, Game-Time, StyleSet, URO-Bench, VocalBench, VoxDialogue, Multi-Bench) did not cover. It separates two abilities that are usually conflated — retaining an instruction and acting on it — and shows current models have the first but not the second. The attention analysis gives a concrete mechanistic candidate (attention dilution away from global style constraints), and the system-message result challenges the assumption that system prompts reliably govern persistent behavior in speech models. The human validation of the emotion and accent judges also provides evidence that LALM-based judging is viable for accent, and considerably less reliable for emotion.

Real-world applications:

• Voice assistants and conversational agents that users configure once with a preferred voice persona and expect to remain stable over a long session.
• Emotional support, tutoring, or storytelling systems where a sustained sad, calm, or fast delivery is part of the intended user experience rather than decoration.
• Accessibility and localization use cases that depend on a consistently maintained accent, volume, or speaking rate for comprehension.
• Speech-to-speech translation and dubbing pipelines where a target style must persist across a multi-turn exchange.

Industry relevance. The finding that proprietary models recall the instruction but fail to express it points to a generation-time control problem rather than a context-length problem, which matters for anyone building long-session voice products. The recall workaround is cheap to implement at inference time, and the system-message degradation is a practical warning for teams that standardize on system prompts for persona configuration. Because the effect appears across three architecturally distinct designs — the thinker-talker design of Qwen2.5-Omni, the interleaved audio-text approach of Step-Audio 2 mini, and the full-duplex streaming of Gemini Live — it looks like a shared problem rather than a quirk of one training recipe.

Future Directions

• Composite and more realistic style settings. The current framework evaluates one speaking style at a time, and the authors state they did not compose multiple styles in a single instruction because models cannot reliably follow even one; they explicitly leave composition as future work, with only a preliminary composite experiment (Indian English + Slow, Fast + Anger) on Gemini Live and GPT-4o.
• Role-playing evaluation. The authors could not assess speech role-playing behaviors at scale because no reliable automatic judges exist for them, and identify this as a practically important scenario requiring consistency across turns.
• Training-data attribution. Since most SLMs do not disclose their training data composition, the paper cannot investigate how data affects style amnesia, and the attention analysis is limited to open-source models.
• Improving style adherence at generation time. The paper frames closing the gap between style retention and stylistic control — rather than improving memorization — as the key direction, noting that progress will require advances in both SLMs and judge models.

Target Audience

Researchers and engineers working on spoken language models, expressive or paralinguistic speech generation, and multi-turn dialogue evaluation will benefit most. The paper is also useful for practitioners designing long-session voice agents who need to know how reliable persistent style configuration actually is, and for evaluation researchers interested in the reliability of automatic speech judges, since the paper reports a human-validation study with Cohen's Kappa and MCC figures for both accent and emotion judges.

Authors’ abstract

In this paper, we show that when spoken language models (SLMs) are instructed to speak in a specific speaking style at the beginning of a multi-turn conversation, they cannot maintain the required speaking styles after several turns of interaction; we refer to this as the style amnesia of SLMs. We focus on paralinguistic speaking styles, including emotion, accent, volume, and speaking speed. We evaluate three proprietary and two open-source SLMs, demonstrating that none of these models can maintain a consistent speaking style when instructed to do so. We further show that while SLMs can recall the style instruction when prompted in later turns, they still fail to express it, but through explicit recall can mitigate style amnesia. In addition, SLMs struggle more when the style instruction is placed in system messages rather than user messages, even though system messages are specifically designed to provide persistent, conversation-level instructions. Our findings highlight a systematic gap in current SLMs' ability to maintain speaking styles, highlighting the need for improved style adherence in future models. Our code and evaluation data are publicly available at https://github.com/YuXiangLin1234/SLM-Style-Amnesia.