MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

AI summary

Authors: Sangyun Chung, Se Yeon Kim, Youngchae Chee, Yong Man Ro (Integrated Vision Language Lab, KAIST, South Korea) arXiv: 2601.21181v2 [cs.AI], 11 Sep 2026 Code: https://github.com/top-yun/MAD

Overview

Research area: Multimodal large language models (MLLMs), specifically hallucination mitigation in audio-visual language models via training-free inference-time decoding.

Technical level: Intermediate. The paper assumes familiarity with autoregressive decoding, logits, softmax normalization, and contrastive decoding, but the core idea is describable without deep mathematics.

Scope (one sentence): The paper proposes Modality-Adaptive Decoding (MAD), a training-free decoding method that lets a multimodal model self-assess which modalities a question requires and then scales contrastive decoding branch strengths accordingly to reduce cross-modal hallucinations.

What This Paper Is About

Multimodal models that take video, audio, and text sometimes let one modality improperly influence what they say about another — for example, describing sounds that are not in the audio because the video suggested them, or inventing visual events because of what was heard. Existing contrastive decoding methods for multimodal models apply the same fixed distortion to all modalities without checking which modality the question actually depends on. MAD addresses this by having the model itself report which modalities it needs for a given question, then weighting modality-specific contrastive branches by those self-assessed relevance scores.

Key Contributions

1. A training-free, modality-adaptive decoding method that dynamically determines the modality requirements of each task and adjusts contrastive decoding accordingly to mitigate cross-modal hallucinations in audio-visual LLMs.
2. A task-driven modality weighting scheme that extracts explicit modality preferences by querying the model ("which modality is needed: audio, video, or both?"), converts the corresponding logits into normalized weights via softmax, and uses them for adaptive fusion of modality-specific contrastive distributions.
3. A four-branch weighted contrastive formulation that combines visual contrastive decoding when audio is present, audio contrastive decoding when visual is present, and single-modality contrastive decoding when one modality is absent, each scaled by a shared base strength γ multiplied by its modality-specific weight.
4. Empirical validation across two cross-modal hallucination benchmarks (CMM and AVHBench), reporting improvements over the base model, VCD-Extended, and AVCD, while maintaining or slightly improving performance on general audio-visual QA benchmarks.

Main Findings

• Headline improvements: On CMM and AVHBench, MAD reports 7.8% and 2.0% improvements for VideoLLaMA2-AV, and 8.7% and 4.7% improvements for Qwen2.5-Omni, as stated in the abstract.
• CMM overall accuracy: VideoLLaMA2-AV improves from 73.5 (base) to 81.3 with MAD, compared to 76.4 with VCD-Extended and 75.8 with AVCD. Qwen2.5-Omni-7B improves from 72.7 to 81.4 with MAD, compared to 72.8 with VCD-Extended and 73.3 with AVCD.
• CMM per-category results (VideoLLaMA2-AV, base → MAD): Visual Dominance 71.8 → 82.3, Audio Dominance 80.0 → 84.3, Language Dominance 68.8 → 77.5.
• CMM per-category results (Qwen2.5-Omni-7B, base → MAD): Visual Dominance 64.5 → 76.8, Audio Dominance 72.3 → 84.3, Language Dominance 81.3 → 83.3. The paper reports the visual-dominance gain as +12.3% and the audio-dominance gain as +12.0% in the text.
• AVHBench results (VideoLLaMA2-AV, base → MAD): Video-Driven Audio Hallucination 75.7 → 79.7, Audio-Driven Video Hallucination 79.0 → 79.1, Overall Accuracy 77.4 → 79.4.
• AVHBench results (Qwen2.5-Omni-7B, base → MAD): Video-Driven Audio Hallucination 73.0 → 78.7, Audio-Driven Video Hallucination 80.7 → 84.4, Overall Accuracy 76.9 → 81.6.
• Self-assessed weights align with intuition: Using 100 videos randomly sampled from VideoMME and 300 constructed questions across three types (visual-related, audio-related, audio-visual-related), the extracted weights show dominant w_v for visual questions, dominant w_a for audio questions, and dominant w_av with a balanced mix of w_v and w_a for audio-visual questions — without any additional supervision.
• Adaptive weighting beats simpler fusion schemes: On CMM with VideoLLaMA2-AV, weighted fusion reaches 81.3 overall accuracy versus 79.4 for uniform weighting (w_av = w_v = w_a = 1/3) and 78.7 for argmax weighting, and versus 73.5 for the baseline.
• Every modality weight helps: Ablating any single weight degrades CMM overall accuracy relative to using all three (81.3). Removing w_a gives 78.0 (with the paper reporting a 6.5% decrease in Visual Dominance), removing w_v gives 78.3 (with a 3.0% decline in Audio Dominance), and using w_a + w_v without w_av gives 78.9.
• General audio-visual QA is not harmed: On VideoLLaMA2-AV, MAD gives OmniBench 36.8 versus 36.3, Worldsense 25.6 versus 23.3, and Music-AVQA 79.1 versus 78.1.
• Hyperparameter choice: The base contrastive strength γ was tuned by sampling 100 examples per dataset and varying γ from 0.5 to 3.0 in intervals of 0.5; γ = 2.5 was used for all datasets, with temperature set to 0 for deterministic generation.

Methodology in Plain English

The starting point is contrastive decoding: you run the model twice, once with the real input and once with a deliberately corrupted version of it, and subtract the corrupted run's logits from the clean run's logits. Tokens that genuinely depend on the real input show a large gap and survive; tokens that come from language priors or spurious correlations look similar in both runs and get suppressed.

MAD extends this to two modalities (video and audio) by building four contrastive branches: a visual contrastive branch evaluated while audio is present, an audio contrastive branch evaluated while visual is present, a visual branch evaluated while audio is absent, and an audio branch evaluated while visual is absent. Each branch targets hallucinations caused by a particular modality configuration.

The novelty is how strongly each branch is applied. Rather than a fixed strength, each branch's strength is γ times a weight w_m. To get those weights, the researchers append a fixed query prompt to the input asking the model which modality is needed — audio, video, or both — and read the logits for the tokens 'both', 'video', and 'audio'. A softmax over those three logits yields normalized weights (w_av, w_v, w_a). A question about sound therefore raises the audio weight and sharpens audio-side suppression, while a question about color sharpens the visual side. All branches are combined softly according to their estimated relevance, and the next token is chosen by taking the argmax of the resulting MAD logits.

The method requires no retraining, no extra annotations, and no separate predictor network — it reuses the model's own next-token prediction machinery for the self-assessment step. Comparisons are made against VCD-Extended, which contrasts against all possible modality distortions with a single shared strength, and AVCD, which applies uniform distortion without query-specific adaptation.

Why This Matters

The paper reframes cross-modal hallucination not as a representation defect within one modality, but as a failure of modality-interaction control — deciding which inputs to trust, suppressing misleading ones, and preserving modality boundaries. It shows that a model can already report its own modality needs reliably enough to drive decoding, which is a cheap and interpretable signal available at inference time.

Potential real-world applications (grounded in the paper's motivating tasks):

• Video question answering, where answers must be grounded in the correct stream rather than a salient but irrelevant one.
• Audio-visual scene understanding, where descriptions must not invent sounds or visual events that were never present.
• Assistive narration and audio description systems, where fabricated audio or visual events would misrepresent the actual content to a user who cannot see or hear it.
• Multi-sensor and multimodal monitoring pipelines, since the authors indicate the framework is intended to extend to combinations such as thermal–RGB and other multi-sensor settings.

Industry relevance: Because MAD is training-free and adds no parameters, it can be dropped onto existing audio-visual LLM deployments as an inference-time wrapper, which lowers the cost of improving reliability compared with retraining or fine-tuning. The self-assessment step also produces an inspectable modality weighting, which is useful for debugging and auditing multimodal systems.

Future Directions

• Learn a lightweight, parameter-efficient predictor for modality weights instead of extracting them directly from the underlying MLLM, so weights can be estimated faster and more accurately.
• Extend the modality-adaptive framework beyond audio–video to richer modality combinations, including thermal–RGB and other multi-sensor settings.
• Investigate the bidirectional nature of cross-modal hallucination further, since the ablations indicate visual dominance induces audio hallucinations and audio dominance induces visual ones — an open question is whether a single weighting scheme can be made fully symmetric across all modality pairs.
• Broaden evaluation, since the experiments cover CMM, AVHBench, OmniBench, Worldsense, and Music-AVQA, and the paper's setup text refers to three audio-visual LLMs while listing two (VideoLLaMA2-AV-7B and Qwen2.5-Omni-7B); dataset sizes for CMM and AVHBench are not reported in the provided content, and no claims are made about scaling behaviour across larger or non-audio-visual model families.

Target Audience

Researchers and engineers working on multimodal large language models who need hallucination mitigation without retraining; practitioners deploying audio-visual systems who want an inference-time reliability improvement; and readers interested in inference-time decoding strategies, contrastive decoding, or modality-fusion control. Prior exposure to autoregressive decoding and logits helps, but the central idea is accessible to those with a general machine learning background.

Authors’ abstract

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8\% and 2.0\% improvements for VideoLLaMA2-AV, 8.7\% and 4.7\% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at \href{https://github.com/top-yun/MAD}{https://github.com/top-yun/MAD}