Resurfacing Paralinguistic Awareness in Large Audio Language Models

AI summary

Overview

Research area: Speech and audio-language modelling — specifically Large Audio Language Models (LALMs), layer-wise interpretability, paralinguistic (non-verbal) cues in speech, parameter-efficient fine-tuning, and AI safety for child users. The paper is listed under cs.SD and appears on arXiv as 2603.11947v1.

Technical level: Intermediate. The paper assumes familiarity with transformer layer representations, linear probing, supervised fine-tuning, LoRA adaptors, and evaluation with an LLM judge, but the core argument is stated in plain terms.

Scope: The paper diagnoses why LALMs ignore paralinguistic cues, identifies which transformer layers carry those cues versus semantic content, and proposes a selective fine-tuning protocol to restore paralinguistic awareness across age, gender, and emotion.

What This Paper Is About

Large Audio Language Models understand what a user says but largely ignore how they say it, discarding cues such as the speaker's age, gender, and emotional state. This matters because the same sentence ("it's raining again today", or "how do I fix my broken lamp?") calls for very different responses depending on who is speaking — and a response that is fine for an adult may be dangerous for a child. The paper's goal is to figure out why this awareness disappears inside the model and to design a fine-tuning method that brings it back efficiently, without retraining every parameter.

Key Contributions

1. The first framing of child safety as a paralinguistic-awareness problem in LALMs. The authors define seven child-safety topics and manually construct a dataset around the failure mode where a model gives an adult-level, step-by-step answer to a child user asking about an activity that is risky without supervision.
2. Five diverse layer-wise analyses that jointly identify paralinguistic layers (layers 0–6) and semantic understanding layers (layers 7–14) in a 28-layer LALM, using linear probing on three paralinguistic categories, intent classification (IC) probing on Fluent Speech Commands, IC cosine similarity, age-aware cosine similarity, and logit lens.
3. A paralinguistic-enhanced fine-tuning (PE-FT) protocol consisting of selective-layer fine-tuning (jointly tuning layers 0–14 while freezing the rest) and an auxiliary dual-level classification head (ADCH) that predicts a primary paralinguistic category label and a secondary speaker-attribute label from the layer-14 representation.
4. Two new evaluation metrics, PA-score and PA-rate, which use a GPT-4.1 judge to score whether a response actually reflects the user's paralinguistic attribute (1), gives a generic response (0), or reflects the wrong attribute (−1). The authors motivate these as a standard, discriminative paradigm that the audio-centred ParaS2S score does not provide.

Main Findings

• Paralinguistic signals live in the early layers, and semantics in the middle layers. Probing Qwen2.5-Omni and Kimi-Audio across 28 layers (0-indexed) showed consistently high, stable accuracy for age, gender, and emotion classification in layers 0–6, followed by a sharp drop at layer 7. Intent classification (IC) probing showed the opposite pattern: low accuracy in layers 0–6, a sharp rise at layer 7, and a clear decrease from layer 15. The authors attribute this synchronous switch to the content-centred modality-tuning protocol, which suppresses paralinguistic signals in favour of speaker-invariant, semantic representations.
• Two further analyses corroborate the layer boundary. The cosine similarity difference Δ(l) between within-intent and cross-intent similarity jumps at layer 7 and stays high until layer 14. In the age-aware analysis, cross-group age pairs (e.g., 6 vs. 30) drop in similarity at layer 7 and reach a local minimum at layer 14, whereas within-group pairs (6 vs. 7, 29 vs. 30) stay high and nearly flat across all layers. In the non-age-declaration condition (child vs. adult speakers), similarity increases across layers 7–14 — evidence that the models suppress paralinguistic signals rather than incorporating speaker age into meaning.
• Deep layers form query-specific features for token prediction, not paralinguistic ones. Logit lens showed Qwen2.5-Omni reaching 40% accuracy in the middle layers and rising from layer 21 onward toward 100%, while Kimi-Audio shows non-zero accuracy from layer 19, rising sharply at layer 21 and quickly approaching 100%. The authors use this to exclude those layers as irrelevant for paralinguistic awareness.
• Selective tuning of layers 0–14 beats full-model tuning. For Qwen2.5-Omni, full-layer tuning (0–27, no ADCH) reached PA-scores of 0.915 (age), 0.955 (gender), and only 0.393 (emotion), while layers 0–14 without ADCH reached 0.960, 0.945, and 0.460. For Kimi-Audio, full-layer tuning reached 0.890 on age, while the selective setting reached 0.915 on age and 0.970 on gender.
• The full PE-FT setting gives the best overall trade-off. With ADCH at layer 14, Qwen2.5-Omni reached PA-scores of 0.945 (age), 0.965 (gender), 0.503 (emotion), and PA-rates of 97.3, 96.5, and 74.5. Kimi-Audio reached 0.940, 0.965, 0.625 with PA-rates of 97.0, 96.5, 80.5.
• Vanilla models are near chance. Qwen2.5-Omni's vanilla PA-scores were 0.010 (age), 0.100 (gender), 0.015 (emotion); Kimi-Audio's were −0.005, 0.085, and 0.018. PA-rates were 50.5, 15.0, and 31.3 for Qwen2.5-Omni and 49.8, 12.5, and 37.3 for Kimi-Audio.
• Layer-range ablations confirm the choice. Fine-tuning deep layers failed: Qwen2.5-Omni's layers 15–27 gave 0.745 (age), 0.745 (gender), 0.018 (emotion), and layers 20–27 gave 0.575, 0.590, −0.080. For Kimi-Audio, deep-layer tuning was worse than the vanilla model on all three categories (layers 20–27: −0.040, −0.045, −0.240).
• ADCH helps most on the hardest category, emotion. Adding ADCH improved Qwen2.5-Omni's emotion PA-score from 0.393 to 0.482 at layers 0–27 and from 0.368 to 0.430 at layers 7–14; for Kimi-Audio, from 0.623 to 0.650 and from 0.310 to 0.433. Placing ADCH at layer 14 was competitive or best in every category, with the largest emotion gains.
• Child-safety behaviour improves dramatically and generalises. Vanilla Qwen2.5-Omni and Kimi-Audio scored PA-rates of 7.14% and 4.29% on the child-speaker child-safety samples. After PE-FT they reached 97.14% and 98.57% — even though those samples were not in the PE-FT training set.
• General capability holds up better under PE-FT. All fine-tuned models showed a slight decline on the VoiceBench helpfulness score relative to vanilla (Qwen2.5-Omni: 73.42 vanilla, 71.16 full-layer, 71.82 selective, 72.34 PE-FT; Kimi-Audio: 76.91, 75.77, 75.22, 76.06). PE-FT degraded it least.
• The representation space reshapes. t-SNE at layer 14 shows vanilla models forming three coarse clusters by paralinguistic category but with different speaker attributes heavily intermixed within each cluster; PE-FT models form clearer sub-clusters by paralinguistic attribute.

Methodology in Plain English

The authors start by asking a diagnostic question: if paralinguistic information is present in the speech signal, which parts of the model actually hold it? To find out, they run audio through Qwen2.5-Omni and Kimi-Audio and, at every one of the 28 transformer layers, average the audio hidden states to get a single vector per layer. They then train small linear classifiers on those vectors and see how well each layer's representation supports three paralinguistic classification tasks (age, gender, emotion) with 100 samples per attribute, repeating the sampling three times.

They run the same trick for semantic understanding using intent classification on Fluent Speech Commands, sampling 1% of the dataset as the train split and repeating the re-splitting five times. They add two complementary analyses: a cosine-similarity measure contrasting near-identical prompts with opposite intents, and an age-aware analysis in which each child-safety query is re-synthesised in four age-declared variants ("I'm six years old", "I'm seven years old", "I'm twenty-nine years old", "I'm thirty years old"). Finally, logit lens checks which layers are already doing next-token prediction, so those can be excluded as "not about paralinguistics".

The picture that emerges is a two-stage model: layers 0–6 hold paralinguistic signals, layers 7–14 build semantic understanding, and the switch at layer 7 is where the model discards the speaker and keeps the words. PE-FT acts on that diagnosis. It fine-tunes only layers 0–14 so the early paralinguistic signal can flow into the semantic stage, freezes everything else, and adds ADCH — a small auxiliary head that, during training only, must predict which paralinguistic category a query depends on and which speaker attribute was used. ADCH is discarded at inference. Training uses a LoRA adaptor on the LLM modules with the audio encoder frozen, 10 epochs, batch size 128, learning rate 8e-5, two A100 GPUs, and about 70 minutes of fine-tuning; the ADCH weight λ is set to 0.5.

The training data comes from GPT-4.1-generated text: 1500 samples per paralinguistic category (age, gender, emotion), each rendered into two audio recordings with contrasting speaker attributes, giving 9000 audio recordings. Age audio comes from Typecast TTS (11 child speakers, 11 adult speakers), gender from Google TTS (10 female, 9 male speakers), and emotion from gpt-4o-mini-tts-2025-03-20 under the instruction "speak in a very {emotion state} tone" across 13 speakers and six states (happy, surprised, sad, angry, disgusted, fearful). Evaluation uses 200 human-annotated text samples per category, each with two audio realisations, for 1200 recordings total. The child-safety set adds seven scenarios (Electrical, Kitchen, Height, Fire, Pet, Medicine, Chemicals) with 10 samples each, 70 in total, synthesised with child and adult voices.

Why This Matters

Impact on research. The paper reframes an architectural finding — that content-centred tuning actively erases speaker information — as a safety and interaction problem. It provides a reusable diagnostic recipe (three paralinguistic probes, two semantic probes, logit lens) and two metrics (PA-score, PA-rate) that separate general response quality from paralinguistic responsiveness. The observation that ParaS2S scores were largely consistent across all three fine-tuning settings, while PA-scores diverged sharply, is the paper's sharpest argument for why discriminative metrics are needed.

Real-world applications:

• Child-facing voice assistants. The child-safety evaluation shows vanilla models answering risky queries (electrical repair, kitchen knife use, medication dosing, chemicals) with adult-level guidance at PA-rates of 7.14% and 4.29%, and reaching 97.14% and 98.57% after PE-FT without topic-specific training.
• Empathetic conversational agents. A user saying "it's raining again today" should be answered differently depending on whether they sound happy or sad; PE-FT raises emotion PA-rate on Qwen2.5-Omni from 31.3 to 74.5 and on Kimi-Audio from 37.3 to 80.5.
• Voice-based customer service and contact centres. Routing and scripting that depends on the caller's apparent age, gender, or emotional state could be made implicit in the audio rather than requiring the user to declare it.
• Accessibility and human-computer interaction. Speech interfaces that adapt tone and content to the speaker's context, rather than treating every user as an undifferentiated text input.

Industry relevance. The method is deliberately cheap: it trains only 15 of 28 layers, uses LoRA with a frozen audio encoder, discards the auxiliary head at inference, and completes in about 70 minutes on two A100 GPUs. Because it does not require a trainable audio synthesiser — the stated limitation of the reinforcement-learning-based ParaS2S approach — it applies to LALMs like Qwen2.5-Omni that the earlier method could not cover. For teams deploying speech assistants, that combination of low compute and preserved general capability (the smallest VoiceBench helpfulness drop of the three settings) is directly practical.

Future Directions

• Broaden the paralinguistic attributes. The work covers three categories (age, gender, emotion). Whether the layer 0–6 / 7–14 split and the PE-FT recipe extend to other attributes — accent, health state, speaking style — is untested

Authors’ abstract

Large Audio Language Models (LALMs) have expanded the interaction with human to speech modality, which introduces great interactive potential, due to the paralinguistic cues implicitly indicating the user context. However, building on the current content-centred paradigm, LALMs usually neglect such paralinguistic cues and respond solely based on query content. In this work, to resurface the paralinguistic awareness in LALMs, we introduce five diverse layer-wise analyses to jointly identify paralinguistic layers and semantic understanding layers. Based on these insights, we propose a paralinguistic-enhanced fine-tuning (PE-FT) protocol accordingly to equip LALMs with paralinguistic-aware capabilities, including (1) selective-layer fine-tuning, and (2) an auxiliary dual-level classification head. Our experiments demonstrate that PE-FT protocol efficiently and effectively resurfaces the paralinguistic awareness, even surpassing the performance of the all-layer fine-tuning strategy.