When Audio-LLMs Don't Listen: A Cross-Linguistic Study of Modality Arbitration

AI summary

Overview

• Research area: Multimodal natural language processing — specifically how speech-enabled large language models (audio-LLMs) arbitrate between conflicting spoken audio and text inputs, studied across languages.
• Technical level: Intermediate. The experimental design is conceptually simple (forced-choice questions), but the paper assumes familiarity with LLM evaluation, ASR, LoRA fine-tuning, and modality-conflict literature.
• One-sentence scope: The paper introduces ALME, a 57,602-stimulus multilingual benchmark of controlled audio-text conflicts, and the Text Dominance Ratio (TDR) metric, to measure how often four audio-LLMs follow conflicting text when explicitly instructed to follow what they hear.

What This Paper Is About

Speech-enabled LLMs are typically given text context (system prompts, conversation history, retrieved documents) alongside spoken input. When that text contradicts what the user actually said aloud, the model must decide which source to trust. This paper asks a simple question — when audio and text conflict, which modality does the model follow? — and shows that models follow text far more often than they should, even when explicitly told to trust the audio.

Key Contributions

1. ALME and TDR: A benchmark of 57,602 controlled audio-text conflict stimuli across eight languages (English, German, French, Italian, Portuguese, Arabic, Japanese, Chinese; four scripts, five language families), plus the Text Dominance Ratio (TDR) as the primary metric — the proportion of conflict trials where a model follows text over audio.
2. The audio-text arbitration gap: A persistent preference for conflicting text over audio across both commercial and open-source models, which is not explained by transcript quality alone and which cannot be reduced to a general instruction-following failure (established via a cascade baseline).
3. Prompt-based mitigation: Framing the conflicting transcript as deliberately corrupted reduces TDR by 80% (from 19% to 3.8%), whereas forcing explicit transcription before answering increases TDR by 14% (to 33%) — the opposite of what a comprehension-based explanation would predict.
4. Fine-tuning ablation: Adapter-only tuning of the audio projection layer increases TDR (+26.5), while LLM LoRA reduces it (−23.9), suggesting arbitration behavior depends more on the LLM's conflict-resolution dynamics than on the audio input path alone.

Main Findings

• Text dominance is large and consistent: Under audio-text conflict, Gemini 2.0 Flash shows TDR of 16.6% and GPT-4o 23.2%. Ultravox v0.6-llama-3.1-8b reaches 48.8% and Qwen2-Audio-7B-Instruct 63.2% — a 47% spread between the lowest and highest model (Cohen's h = 1.01).
• The effect is modality-specific, not a general instruction-following failure: Replacing audio with its Whisper transcript under identical reliability cues drops TDR to 1.6% for Gemini and 0.9% for GPT-4o (Cascade-B). Cascade-C (reference transcript) gives 1.0% and 0.7%. Effect sizes for the multimodal-vs-cascade gap: h = 0.59 for Gemini, h = 0.82 for GPT-4o.
• Text-text and audio-text arbitration are independent capabilities: GPT-4o is better than Gemini at text-text arbitration (0.9% vs. 1.6%) yet worse at audio-text arbitration (23.2% vs. 16.6%).
• Audio comprehension quality does not explain text dominance: Gemini's audio-only accuracy (97.2%) exceeds its cascade accuracy (93.9%), yet its TDR is still 16.6%. GPT-4o's audio-only accuracy (92.7%) is worse than Whisper ASR (96.2%), yet it shows an even larger multimodal-to-cascade gap.
• Substantial cross-linguistic variation: Gemini shows roughly 4× higher TDR for Chinese (31.8%) than English (8.1%); GPT-4o shows roughly 3.8× (42.8% vs. 11.3%). Qwen2-Audio shows the reversed pattern, with lower TDR for CJK/Arabic languages (50–56%) than European languages (64–71%). All χ² tests (df = 7) yield p < 0.001.
• Reducing text trust works; forcing audio attention does not: Adversarial framing ("deliberately corrupted") reduces TDR from 19% to 3.8% (h = −0.51); explicit-ignore ("COMPLETELY IGNORE the transcript") reaches 7.9% (h = −0.33); audio-first prompting increases TDR to 33% (h = +0.32). Adversarial framing cuts Japanese TDR from 30.2% to 5.7%, substantially closing the English-Japanese gap.
• Fine-tuning the LLM helps, fine-tuning the adapter hurts: On held-out flip types (n = 29,263), Ultravox's baseline TDR of 49.4% rises to 75.9% with adapter-only tuning (+26.5) and falls to 25.5% with LLM LoRA (−23.9). Both conditions drop audio-only accuracy by roughly 25 percentage points (to 53.8% and 56.3%) while preserving aligned accuracy (84.5% and 86.2%).
• LoRA mitigation is language-dependent: LLM LoRA brings European languages near Gemini-level TDR (English 37.4% → 7.6%, German 48.5% → 10.0%, Italian 44.1% → 9.1%) but leaves CJK/Arabic at 49–54%.
• TTS resynthesis shifts open-source models toward text: GPT-4o decreases by 4.2% and Gemini changes by +0.2% (not significant), while Ultravox increases by 3.5% and Qwen2 by 2.2% (both p < 0.001).
• Negation addition produces the highest TDR across all four models (e.g., Gemini 29.2%, GPT-4o 37.0%), and Qwen2-Audio shows severe position bias (27.6% gap), though position-corrected TDR (62.6%) matches overall TDR (62.7%).

Methodology in Plain English

Each test item presents a model with three things: a short clip of natural speech from Common Voice v22.0 (1.5–8 seconds, 3–25 word transcripts), a modified transcript in which exactly one semantic element has been changed, and a binary forced-choice question about that element. If the model answers with the audio's version, it followed audio; if it picks the transcript's version, it followed text. Four conditions are used: audio-only, text-only, aligned (both correct), and conflict (the primary condition). In the conflict condition the prompt explicitly warns that the transcript may be incorrect and instructs the model to answer based on what it HEARS.

The four flippable element types are number swaps, negation insertion/removal, adjective swaps, and time expression swaps, with per-language dictionaries of 6–65 entries per category. Questions were generated by GPT-4.1-mini with randomized answer order and passed through an additional LLM review. Native speakers validated 40 randomly sampled stimuli each for English, Japanese, and Portuguese, with 97.5%, 92.5%, and 90% judged valid respectively under a forced-choice discriminability criterion.

To separate modality effects from general instruction-following limitations, the authors built a cascade baseline in which Whisper large-v3 transcribes the audio and each API model answers with two competing text sources under identical reliability cues. Three cascade conditions are used (Cascade-A: ASR transcript only; Cascade-B: ASR plus conflicting text, n = 53,747; Cascade-C: reference transcript plus conflicting text). Prompt interventions were tested on Gemini over 14,369 English and Japanese stimuli. All 57,602 stimuli were resynthesized with Azure Neural TTS (six neural voices per language) as a robustness check, and Ultravox was fine-tuned under adapter-only and LLM LoRA (rank 16, α = 32) conditions. All evaluations used temperature = 0 and structured JSON output; parse failure rates were below 1% for all models. Statistics use Wilson score 95% confidence intervals plus speaker-clustered bootstrap CIs (1,000 iterations over 20,444 unique speakers), with Bonferroni correction (α_adj = 0.0018) for pairwise language comparisons.

Why This Matters

The paper reframes text dominance in audio-LLMs as a failure of arbitration accessibility — models can hear fine, but cannot use what they hear when text disagrees — rather than a comprehension failure. This matters because existing evaluations measure transcription accuracy or task performance under aligned inputs, implicitly assuming that once speech is understood, downstream reasoning is unaffected by input modality. The results show that assumption is wrong.

Real-world applications:

• Voice interfaces and assistants: Deploying a model with high TDR means users whose spoken corrections contradict stale conversation history may be ignored.
• Multilingual speech systems: CJK and Arabic speakers may experience systematically lower audio fidelity in speech-based interactions, since three of four models show 2–4× higher TDR for those languages.
• System prompt design: Framing accompanying text as potentially corrupted reduces TDR by 80% with no model modification — immediately deployable.
• Model selection: Choosing among audio-LLMs implicitly chooses an arbitration policy; the 47% TDR spread means this choice has large behavioral consequences.

Industry relevance: The paper connects to documented ASR performance disparities across language groups and argues that model selection encodes a policy on how conflicting inputs are resolved. For any deployment where audio should be authoritative (voice interfaces, transcription correction), lower-TDR models are preferable, and English-only benchmarks are insufficient for predicting multilingual behavior.

Future Directions

• Open-ended generation: The forced-choice format makes arbitration directly observable but does not capture free-form interaction, where reliance on audio or text is harder to attribute. New attribution methods are needed.
• Symmetric analysis: The paper measures TDR (following text when told to follow audio). A complementary Audio Dominance Ratio — following audio when told to follow text — could reveal additional patterns.
• Prompt stability: TDR varies from 3.8% under adversarial framing to 33% under audio-first prompting; a systematic stability analysis with minor rephrasings of the baseline conflict prompt was not conducted.
• Broadening the ablation: The fine-tuning ablation was run on Ultravox only due to resource constraints, and both fine-tuned conditions degraded audio-only accuracy by roughly 25 percentage points, indicating a comprehension–arbitration trade-off that needs investigation with larger training sets and more models.

Target Audience

Researchers and engineers working on speech-enabled LLMs, multimodal evaluation, and modality-conflict or knowledge-conflict problems; practitioners deploying voice interfaces and multilingual speech systems who need to reason about which modality a model actually trusts; and teams building benchmarks for audio-LLM reliability. Readers interested in fairness across languages in speech systems will also find the cross-linguistic results relevant.

Authors’ abstract

When audio and text conflict, speech-enabled language models follow text far more often than they do when arbitrating between two conflicting text sources, even under explicit instructions to trust the audio. We introduce ALME (Audio-LLM Modality Evaluation), a dataset of 57,602 controlled audio-text conflict stimuli across eight languages, together with Text Dominance Ratio (TDR), which measures how often a model follows conflicting text when instructed to follow audio. Gemini 2.0 Flash and GPT-4o show TDR 10--26$\times$ higher than a baseline that replaces audio with its transcript under otherwise identical conditions (Gemini 2.0 Flash: 16.6% vs. 1.6%; GPT-4o: 23.2% vs. 0.9%). These results suggest that text dominance reflects not only information content, but also an asymmetry in arbitration accessibility, i.e., how easily the model can use competing representations at decision time. Framing the transcript as deliberately corrupted reduces TDR by 80%, whereas forcing explicit transcription increases it by 14%. A fine-tuning ablation further suggests that arbitration behavior depends more on LLM reasoning than on the audio input path alone. Across four audio-LLMs, we observe the same qualitative pattern with substantial cross-model and cross-linguistic variation.