Text Dominance in Paralinguistic Conflict: Tracing Suppression Through Audio LLM Circuits

AI summary

Overview

Research area: Mechanistic interpretability of audio-capable large language models, specifically how these models weigh transcript content against paralinguistic cues (emotion, speaker identity, sarcasm).

Technical level: Advanced. The paper uses probing classifiers, back-patching, and attention knockout, which assume familiarity with causal-intervention methods in interpretability research.

Scope: The paper measures how often six audio LLMs follow mismatched text over conflicting acoustic information under LISTEN-style conditions, and then traces the internal pathway that produces this text dominance in the most affected model.

What This Paper Is About

Audio LLMs are only more useful than ordinary speech-to-text pipelines if they actually act on how something is said, not just what the words are. The authors test whether these models default to the transcript when audio and text disagree about emotion, speaker identity, or sarcasm, and then look inside the network to find where the transcript's influence takes hold and whether it can be blocked. The goal is both diagnostic (does text dominance happen, and how widespread is it?) and mechanistic (what internal routing causes it, and can it be interrupted?).

Key Contributions

1. A cross-model measurement of text dominance. The authors evaluate six audio LLMs on conflict items where the audio and the transcript point to different answers, using LISTEN-style conditions, and report a mean text-dominance rate of 0.78 (0.86 when sarcasm items are excluded).

2. Evidence that the failure is not perceptual. In the most text-dominant model screened, linear probing shows speaker identity is still decodable at ceiling at the language model's input, meaning the acoustic information is present and accessible internally even though the model's output ignores it.

3. A causal-intervention analysis of where the conflicting claim propagates. Using attention knockout, the authors show the claim's influence travels through intermediate tokens rather than a single direct edge, and that this intervention partially restores audio-grounded output during open-ended generation.

4. A caution about back-patching and about benchmark format. The authors replicate back-patching, a method reported elsewhere to repair lexical text dominance, and find no measurable effect here — but also find it fails on its own source task in their replication (n=300), so they treat the null as uninformative. They further show the intervention effect appears only in free-form generation and not in closed-set multiple choice. Code and data are released publicly.

Main Findings

• Text dominance is the common case, not an edge case. Across six audio LLMs under mismatched text-plus-audio conditions, the mean text-dominance rate is 0.78. Excluding sarcasm, it rises to 0.86, indicating sarcasm items behave somewhat differently from other paralinguistic cues.

• The audio information is not lost, only overridden. Probing at the language model's input in the most text-dominant model recovers speaker identity at ceiling. The model has access to the paralinguistic signal; the failure sits downstream of perception.

• Back-patching provides no usable evidence either way. The technique, reported to repair lexical text dominance, has no measurable effect in this setting. Because the authors' own replication of it on its source task also fails (n=300), they explicitly decline to interpret the null as proof that the method does not transfer.

• The conflicting claim flows through a relay, not a single edge. Attention knockout that blocks the claim from propagating through intermediate tokens during open-ended generation partially restores audio-grounded output. Blocking the answer position's direct connection to the conflicting claim has no effect, implying the influence is routed indirectly.

• Benchmark format can hide these effects entirely. The effect of the intervention appears only in free-form generation and not in closed-set multiple choice, which suggests evaluations built on multiple-choice items may be systematically blind to this class of intervention and, by extension, to the underlying behaviour.

Methodology in Plain English

The authors construct test items where the spoken audio and the written transcript disagree — for example, the words suggest one emotion while the voice suggests another — and record which signal the model's answer follows. That gives a simple behavioural score of text dominance for each of six models.

To find out why the model follows the text, they look inside it. A probing classifier checks whether information about the speaker is still recoverable from the model's internal state at the point where the language model takes over; if it is, the problem is not that the audio was never encoded. They then try to interrupt the model's internal processing in a targeted way: blocking attention paths between specific positions (attention knockout) to see which connections, if severed, change the output. They also attempt back-patching, which transplants internal activations from one run into another to test causal influence, and separately replicate that method on its original task as a sanity check on their own pipeline. Finally, they compare results under free-form generation against closed-set multiple choice to see whether the answer format changes what the intervention does.

Why This Matters

Impact on research. The paper shifts the question from "do audio LLMs use acoustic cues?" to "where in the network is the acoustic signal suppressed, and can that suppression be interrupted?" It also delivers a methodological warning with consequences beyond this subfield: negative results from a replication whose own positive control fails are uninformative, and multiple-choice evaluation can mask effects that free-form generation reveals. Both points bear on how interpretability claims are validated generally.

Real-world applications:

• Crisis hotlines and emergency triage, where detecting distress in a caller's voice can matter more than the literal words used.
• Customer support and call-centre analytics, where tone conveys satisfaction or escalation risk that transcripts flatten out.
• Voice assistants and conversational agents, which need to respond to how something is said to avoid jarring or unsympathetic replies.
• Accessibility and assistive tools, where speaker identity and emotional state are part of the message being conveyed.

Industry relevance. Teams deploying audio LLMs currently rely heavily on multiple-choice benchmarks to certify that their models "understand" paralinguistic cues. If those benchmarks cannot detect the suppression mechanism or interventions against it, deployment decisions may rest on evaluations that do not test the behaviour that matters in production.

Future Directions

• Targeting the relay rather than the edge. Since blocking the direct connection to the conflicting claim does nothing, follow-up work could characterise the intermediate tokens that carry the influence and design interventions aimed at them specifically.
• Rebuilding evaluation around free-form generation. If closed-set multiple choice is blind to these effects, the field needs generation-based benchmarks for paralinguistic grounding, and a clearer account of why the format changes the result.
• Establishing a working causal-intervention baseline. With back-patching uninformative here, the question of which intervention method reliably attributes text dominance to specific components remains open.
• Testing generality. Whether the relay structure and the text-dominance rates hold across other models, other paralinguistic dimensions beyond emotion, identity and sarcasm, and other languages is not settled by this abstract.

Target Audience

Interpretability researchers studying multimodal and audio LLMs; engineers and evaluation leads responsible for certifying that voice models respond to acoustic cues rather than transcripts alone; and research groups working on paralinguistic or emotion-aware speech systems. Readers without a background in causal interventions will find the behavioural results accessible but the mechanistic sections demanding.

Authors’ abstract

Audio LLMs deployed in settings such as crisis hotlines and customer support are only useful beyond speech-to-text pipelines if they use acoustic cues rather than the transcript alone. We test whether they prioritise transcript content over paralinguistic cues (emotion, speaker identity, sarcasm) and trace the mechanism when they do. Across six audio LLMs under LISTEN-style conflict conditions, the mean text-dominance rate on mismatched text+audio items is 0.78 (0.86 excluding sarcasm). In the most text-dominant screened model, probing shows speaker identity remains linearly decodable at ceiling at the language model's input, so the failure is not perceptual. Back-patching, reported to repair lexical text dominance, has no measurable effect here; but it also fails on its own source task in our replication (n=300), so we treat its null as uninformative rather than as evidence of non-transfer. Attention knockout partially restores audio-grounded output by blocking the claim from propagating through intermediate tokens during open-ended generation, whereas blocking the answer position's direct connection to the conflicting claim has no effect. The claim's influence therefore travels through an intermediate relay rather than a single severable edge. The effect appears only in free-form generation and not in closed-set multiple choice, suggesting that benchmarks relying on that format may be systematically blind to interventions of this kind. Code and data are publicly available at https://github.com/AdvayRoongta/audio-llm-mech-interp.