Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts

AI summary

Overview

• Research area: Natural Language Processing / multimodal and omni-modal large language model evaluation, specifically modality bias under cross-modal conflict.
• Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with multimodal LLMs, cross-modal benchmarks, linear probing of hidden states, and contrastive decoding.
• Scope (1 sentence): The paper introduces Tri-PvP, an 8,000-sample tri-modal conflict benchmark that separates modality bias from evidence-form bias by presenting vision, audio, and text as conflicting signals in which vision and audio each take either perceptual or propositional form, then evaluates five omni-modal LLMs, probes their internal representations, and tests a contrastive-decoding intervention.

What This Paper Is About

Omni-modal large language models (OLLMs) process vision, audio, and text together, yet it is unclear which input stream they trust when those streams disagree. The authors argue that existing tri-modal bias benchmarks contain a hidden confound: they mix two qualitatively different kinds of evidence — perceptual signals (a real photograph or recording) and propositional signals (a declarative claim such as "this is a dog") — unevenly across modalities, so a measured "visual bias" might really be a bias toward perceptual evidence, which the image channel disproportionately supplies. Tri-PvP is built to separate these two factors and measure each cleanly.

Key Contributions

1. Identification of a structural confound in existing tri-modal modality-bias benchmarks, where an unmatched mix of perceptual and propositional evidence entangles modality bias with evidence-form bias — and a benchmark, Tri-PvP, of 8,000 samples with controlled perceptual/propositional configurations across vision and audio to address it.
2. Benchmarking five OLLMs (Qwen2.5-Omni-7B (11B), MiniCPM-o 4.5 (9B), Qwen3-Omni-30B-A3B-Thinking (32B), Gemma 4 E4B (8B), and Gemini 3 Flash), showing a robust image bias that evidence form modulates but never reverses, plus a systematic asymmetry: models favor perceptual evidence in vision but propositional evidence in audio.
3. Representation-level analysis via layer-wise linear probing, showing modality-bias information is already linearly decodable from early hidden-state layers, before any token is generated.
4. An inference-time diagnostic intervention adapting contrastive decoding, which partially reduces image bias while preserving general omni-modal competence on OmniBench, but introduces a residual text bias.

Main Findings

• Image bias dominates. Among the 20 (model × evidence-type) conditions in Figure 3, BIAS_IMAGE is the dominant bias label in 18; the two exceptions both arise in Qwen2.5's Prop_I conditions, where BIAS_TEXT or NO_BIAS becomes dominant. BIAS_IMAGE frequently exceeds 60%.
• Audio bias is the weakest. BIAS_AUDIO is the smallest of the three single-modality biases in 18 of 20 bars, typically below 10%. The two exceptions are both in Gemma4's Prop_A conditions.
• Text bias sits in between, generally falling between image and audio bias.
• Evidence form modulates the two non-text modalities in opposite ways. Most models show stronger image bias under perceptual images (e.g., BIAS_IMAGE on Qwen2.5 is 49.7% under Perc_I-Prop_A but only 12.7% under Prop_I-Prop_A), while audio bias generally rises under propositional audio, most notably in Gemma4 (0.9% under Perc_I-Perc_A to 25.4% under Perc_I-Prop_A).
• A systematic evidence-form asymmetry. Models exhibit a stronger bias toward perceptual evidence in vision but toward propositional evidence in audio. The authors hypothesize this reflects the divergent pretraining objectives of vision encoders (optimized for perceptual image content) and audio encoders (often initialized from ASR-style objectives emphasizing linguistic content recovery from speech).
• Unbiased responses peak under fully propositional conditions. The NO_BIAS rate peaks under Prop_I-Prop_A at 60.1% for Qwen2.5, 13.6% for MiniCPM, 11.6% for Gemini3, 7.2% for Gemma4, and 1.5% for Qwen3, while remaining below 10% in most other bars.
• Modality order produces divergent positional biases but does not overturn the conclusions. Qwen2.5 shows a recency bias, favoring the modality placed immediately before the question across all perceptual/propositional combinations. Gemma4 shows different patterns per modality: image bias peaks when the image is first under Prop_I conditions but last under Perc_I conditions; text is relied on more when placed first; audio bias peaks at the front position (24.3%) only under Prop_I-Prop_A. Averaged over all six orderings, BIAS_IMAGE remains dominant in most conditions for both models.
• Image bias is highly linearly decodable; audio bias is the least. Layer-wise linear probes on Gemma4 and Qwen2.5 show the image-involving probe consistently among the top performers. In Qwen2.5, the image probe reaches approximately 80% peak balanced accuracy under both Perc_I conditions, which correspond to the strongest output-level image bias (approximately 50%). Gemma4's audio probe remains near the random baseline under both Perc_A conditions.
• Bias is decodable early. Most probes exceed chance (50%) within the first two layers. Gemma4 progressively strengthens the bias and peaks in middle-to-late layers; Qwen2.5 reaches near-final accuracy within the first few layers with little refinement thereafter.
• Contrastive decoding partially mitigates image bias. On Gemma4, the largest absolute reductions in BIAS_IMAGE occur in Prop_I-Prop_A (74.9% to 38.8%) and Prop_I-Perc_A (74.4% to 39.6%); the largest NO_BIAS gain is in Prop_I-Prop_A (7.2% to 38.6%).
• CD works better against propositional visual content than perceptual: both Prop_I conditions show larger BIAS_IMAGE reductions and larger NO_BIAS increases than the Perc_I conditions.
• A residual text bias emerges. Text bias increases across all four conditions, ranging from 2.2% in Prop_I-Prop_A to 13.5% in Perc_I-Prop_A. BIAS_AUDIO nevertheless remains the lowest bias category in every condition with CD.
• General competence is largely preserved. On OmniBench, CD attains 37.4% overall accuracy on Gemma4 versus 38.4% for the unmodified decoder.

Methodology in Plain English

The authors built a dataset where every sample contains three mutually distinct, conflicting labels — one carried by the image, one by the audio, one by the text — plus an open-ended question with no modality-specific wording (no "what do you see" or "what do you hear") and no multiple-choice options. The image and audio channels each appear in two forms. Perceptual evidence is a real image or a real recording drawn from existing datasets. Propositional evidence is a declarative sentence, instantiated from manually authored templates and rendered either as text on a plain white background (for images) or synthesized with GPT-4o mini TTS using four voices (for audio). Text is always propositional, since the authors argue a symbolic medium can only convey declarative statements.

Four evidence-type conditions result (Perc_I-Perc_A, Perc_I-Prop_A, Prop_I-Perc_A, Prop_I-Prop_A), constructed as matched counterfactuals: the same label triples and questions are instantiated under all four conditions, so comparisons are made on the same triples. Four domains — animal, emotion, environment, and music — each contribute 500 triples, giving 500 samples per condition, 2,000 samples per domain, and 8,000 in total.

Because no modality is more reliable than another by construction, the authors treat NO_BIAS as the appropriate behavior and measure bias as a deviation from it. Model responses are free-form, so a judge model (GPT-5.4 nano) classifies each response into one of eight mutually exclusive categories: three single-modality biases (BIAS_IMAGE, BIAS_AUDIO, BIAS_TEXT), three dual-modality biases (BIAS_IMAGE_AUDIO, BIAS_IMAGE_TEXT, BIAS_AUDIO_TEXT), HALLUCINATION, and NO_BIAS. The judge was validated by manual author verification on 800 samples (10%), stratified uniformly by model, domain, evidence-type condition, and bias type, reaching 97.1% agreement.

To go beyond outputs, the authors train logistic-regression probes on the hidden state at the last prompt-token position of every layer, decomposing each predicted label into per-modality binary targets, and report balanced accuracy against a 50% chance baseline. Finally, they adapt contrastive decoding on Gemma4 with α = 0.5 and β = 0.1, contrasting the full tri-modal logits against logits from audio-plus-question alone in order to surface the audio evidence the model underweights.

Why This Matters

The paper reframes how modality bias should be measured: without controlling for evidence form, a benchmark cannot tell whether a model trusts vision or trusts perceptual evidence, and the two imply different fixes. The authors also show that bias is present in intermediate representations before any token is generated, which means surface-level prompt or decoding fixes are unlikely to be sufficient — a conclusion they state explicitly by calling for mitigation strategies and representation-level causal analysis beyond surface-level interventions.

Real-world applications the work bears on:

• Personal assistants. OLLMs are increasingly deployed as everyday personal assistants that identify objects in photos or query audio content; a systematic preference for the visual stream can make these systems ignore what a user actually said or played.
• Noisy or inconsistent real-world inputs. Modality bias compromises reliability whenever real-world inputs are noisy, redundant, or in disagreement.
• Security and adversarial misuse. Adversaries could embed harmful content in the modality the model most trusts, so knowing which stream a model privileges has direct security implications.
• Assistive and accessibility tooling. The finding that models under-consult perceptual audio — non-linguistic acoustic cues such as a bark or a meow — points to a specific weakness for applications that depend on environmental sound rather than speech.

Industry relevance: the benchmark ships with the four evidence-type conditions and an automatic judge protocol, and the contrastive-decoding experiment shows a parameter-free way to shift behavior at inference time (at the cost of an extra forward pass), which is directly relevant to teams deploying OLLMs who cannot retrain models.

Future Directions

1. Extend beyond the everyday scale. The benchmark covers only everyday-perception scenarios; it is unclear how models behave under broader epistemic scales such as complex diagrams or scientific data.
2. Test non-neutral instructions. The questions are deliberately bias-neutral to elicit unprompted preference; how that preference shifts under prompts that request enumeration of all sources, demand a single answer, or supply source-reliability cues is left as a distinct research question.
3. Systematically tune and broaden mitigation. The contrastive-decoding experiments are confined to a single model with fixed α and β; a sensitivity analysis over these coefficients and evaluation across a broader set of models remain for future work. The residual text bias and the extra forward-pass cost are open problems.
4. Pursue representation-level causal analysis. Since bias is linearly decodable from early layers but only partially mitigated at the surface, the authors call for more fundamental mitigation strategies and causal analysis of how modality preference is encoded.

Target Audience

Researchers and engineers working on multimodal and omni-modal large language models who need to evaluate cross-modal conflict behavior; benchmark designers who care about experimental confounds in bias measurement; interpretability researchers interested in layer-wise probing of modality preference; and practitioners deploying OLLMs as assistants or in security-sensitive settings where an unexamined preference for one input stream creates reliability and safety risk. The paper is also useful to readers tracking methodology in LLM-as-a-Judge evaluation, since it reports a 97.1% judge agreement rate on 800 manually verified samples.

Authors’ abstract

Omni-modal large language models (OLLMs) jointly process vision, audio, and text, yet their modality bias under cross-modal conflict remains underexplored. Existing benchmarks conflate two distinct forms of evidence within a single modality: perceptual signals (e.g., a photograph or recording of a dog) and propositional signals (e.g., the declarative claim "this is a dog"), such that any measured modality bias is inherently confounded with evidence-form bias, precluding clean attribution to either source. To address this, we introduce Tri-PvP, an 8,000-sample tri-modal conflict benchmark crossing vision, audio, and text, where vision and audio each take perceptual or propositional form. Evaluating five OLLMs, we find robust visual bias across most models and evidence-type conditions. Crucially, we reveal a systematic asymmetry in evidence-form bias: models exhibit a stronger bias toward perceptual signal in vision but propositional in audio. Further analyses via layer-wise linear probing and contrastive decoding reveal that modality bias is already linearly decodable from early representation layers and can only be partially mitigated, calling for mitigation strategies beyond surface-level interventions.