MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

AI summary

Overview

Research area: Omni-modal (audiovisual-language) large language models, post-training alignment, and cross-modal hallucination mitigation.

Technical level: Intermediate. The paper assumes familiarity with Direct Preference Optimization (DPO) and multimodal LLM architectures, but presents its core ideas clearly enough for readers who understand preference-based fine-tuning.

Scope: The paper proposes a modality-aware preference optimization objective — MoD-DPO and its extension MoD-DPO++ — that reduces cross-modal hallucinations in omni LLMs by enforcing invariance to corruption of the prompt-irrelevant modality and sensitivity to corruption of the prompt-relevant modality, plus a language-prior debiasing penalty.

What This Paper Is About

Omni LLMs that process audio, video, and text can hallucinate across modalities — for example, "hearing" sounds suggested by visual cues or "seeing" events suggested by audio — because they over-rely on language priors and learn spurious correlations between modalities during pretraining and alignment. Existing multimodal preference optimization methods do not explicitly decouple modality pathways during optimization, and decoding-time defenses such as Visual Contrastive Decoding operate only post hoc. This paper introduces a DPO-style objective that structurally teaches a model which modality it should listen to (and which it should ignore) for a given query.

Key Contributions

1. MoD-DPO, a preference optimization technique for omni LLMs that adds KL-based regularization terms to the standard DPO objective, enforcing invariance to corruption in the irrelevant modality and sensitivity to corruption in the relevant modality. The authors derive a closed-form optimal policy and a corresponding reward, yielding modality-specific losses for visual prompts (ℒ_MoD^v) and audio prompts (ℒ_MoD^a).

2. A language-prior debiasing (LPD) penalty that is added to the preference optimization reward to suppress hallucination-prone, text-only responses. The penalty is −log π_text(y|x^v), where π_text is a frozen language model; for efficiency the authors use the reference model itself (β used in the baseline is β = 0.1, with β_sens = 0.05, β_inv = 0.02, and γ_LPD = 0.05).

3. A new automatically generated preference dataset with 18,112 samples over 10,854 unique videos. Data generation decouples audiovisual input into separate audio and visual annotations, builds QA pairs for captioning and object/event-presence tasks, and constructs hard-negative rejected responses that inject information from the irrelevant modality.

4. Empirical validation across the AVHBench and Curse of Multi-Modalities (CMM) hallucination benchmarks and general audiovisual, video, and audio benchmarks (DailyOmni, MVBench, MMAU), showing consistent gains over vanilla DPO and OmniDPO under comparable training budgets.

Main Findings

• AVHBench accuracy gains: On Qwen 2.5 Omni, MoD-DPO++ reaches accuracy of 88.19 on audio-driven video hallucination (ADVH), 83.40 on video-driven audio hallucination (VDAH), and 69.68 on audiovisual matching (AVM), compared with the untrained reference at 84.15, 77.38, and 54.69 respectively. MoD-DPO without LPD gives 87.66, 82.48, and 69.07.
• Larger gains on audiovisual matching: The paper reports that MoD-DPO produces gains of up to 27% accuracy relative to the reference models on the audiovisual matching task, where the reference model has very low precision (9.81 for Qwen 2.5 Omni) and very high recall (99.57).
• MiniCPM-O 2.6 improvements: The same trend holds on the 8B MiniCPM-O 2.6 model, where MoD-DPO++ achieves 87.26 (ADVH), 79.49 (VDAH), and 60.66 (AVM), versus 83.36, 74.54, and 54.26 for the untrained reference.
• CMM results: On Qwen 2.5 Omni, MoD-DPO++ reaches overall perception accuracy (pa) 89.2 and hallucination resistance (hr) 87.2, compared with 86.4 and 84.6 for the reference — described as roughly a 3–4% overall gain. For MiniCPM-O 2.6, MoD-DPO++ gives 88.3 pa and 83.6 hr versus 85.6 and 80.4.
• LPD helps most on language dominance: The improvement of MoD-DPO++ over MoD-DPO is significantly larger on the language dominance task than on other CMM tasks, indicating the LPD penalty is doing what it was designed to do.
• Ablation confirms all three components: Disabling sensitivity, invariance, and LPD entirely gives AVHBench accuracy 72.07 and CMM pa 86.4 / hr 84.6, while enabling all three gives 80.42 accuracy and 89.2 pa / 87.2 hr. LPD alone raises CMM hr from 84.6 to 86.3.
• Hard negatives matter: Using a rejected response relevant to the opposite modality outperforms a completely irrelevant rejected response (80.42 vs 77.91 accuracy on AVHBench).
• Corruption method matters: Diffusion-based corruption with 500 steps performs best (80.42 accuracy); 10 steps performs poorly (72.72) because the corrupted input looks too similar to the original and spuriously shifts the KL divergence.
• Mismatched audiovisual contexts help: Training with a mix of matched and mismatched audiovisual contexts gives 80.42 AVHBench accuracy versus 76.69 for matched only.
• Joint audiovisual objectives do not help much: Adding an objective for joint audiovisual tasks yields 80.33 AVHBench accuracy and 89.0 pa / 87.5 hr on CMM — no significant improvement over training with audio and visual tasks alone.
• General capabilities improve: MoD-DPO++ raises Qwen 2.5 Omni from 47.34 to 53.82 average accuracy on DailyOmni, from 69.61 to 71.02 on MVBench, and from 64.62 to 66.33 on MMAU. For MiniCPM-O 2.6, DailyOmni goes from 30.55 to 36.60.
• Attention shifts toward audiovisual tokens: An attention analysis on CMM shows the total attention to audiovisual tokens as a percentage of total input-token attention increases significantly with MoD-DPO++, indicating the model focuses more on non-text input.
• Robustness behavior is as designed: Log-likelihood shift analysis on the AVHBench audio-driven video hallucination task shows MoD-DPO++ produces a larger distributional shift when the relevant modality is corrupted and a smaller shift when the irrelevant modality is corrupted.
• Hyperparameter sensitivity: Increasing β_sens up to 0.1 raises precision and perception accuracy, but recall and hallucination resistance drop sharply beyond 0.1. β_inv above 0.02 significantly degrades performance. Higher γ_LPD sharply reduces precision and perception accuracy, presumably because it suppresses desirable language priors as well.
• Training budget: Because extra forward passes are needed for corrupted audio, corrupted video, and text-only inputs, MoD-DPO++ is trained for only one quarter of the total epochs used by baseline methods (MoD-DPO uses one epoch; baselines use four), with a learning rate of 3e-7, batch size 1 per GPU on 8 H100 GPUs.

Methodology in Plain English

The starting point is Direct Preference Optimization, which aligns a model by comparing a "chosen" and a "rejected" response and pushing the model's policy toward the chosen one while keeping it close to a frozen reference model. The authors modify this in two ways.

First, they split the inputs by modality role. For a prompt about what is happening visually, the audio track is the irrelevant modality. The objective adds a penalty term that asks the model's output distribution to stay the same whether the audio is intact or corrupted (invariance). It also adds a term that asks the output distribution to change when the video itself is corrupted (sensitivity). This mirrors a simple intuition: a question about the visuals should be answered the same way regardless of what the soundtrack says, but should change if the visuals change. A symmetric loss is written for audio-related prompts. The authors derive a closed-form optimal policy for this modified objective by treating the corrupt-input distributions as fixed targets within each step, and they stop gradients when passing corrupted inputs so those distributions do not drift. During training they alternate between batches of vision-related and audio-related prompts.

Second, they add a language-prior debiasing term. Because the underlying LLM is trained on enormous text corpora, it tends to answer from text alone. The LPD penalty subtracts the log-probability that the frozen reference model assigns to the chosen response given only the text prompt, discouraging responses that are plausible from language alone.

Building the data was a major part of the work. In Stage 1, audio and video are separated and annotated independently: GPT-4o produces visual captions, RAM++ produces visual tags, and AudioFlamingo 3 produces audio captions, drawing on MSR-VTT, VALOR32K, and AudioCaps. In Stage 2, GPT-4o generates QA pairs for captioning and object/event-presence tasks. In Stage 3, instead of generating obviously wrong rejected responses, the pipeline creates hard negatives that import content from the wrong modality — exactly the kind of error the hallucination benchmarks test for. The authors also deliberately create mismatched audiovisual contexts by pairing audio and video from different files, which empirically helped.

Why This Matters

Impact on research. The paper reframes hallucination mitigation as a problem of modality role assignment rather than generic alignment. Rather than adding more data or a separate reward model, it encodes two structural priors directly into the preference objective, and derives a closed-form solution that fits into the existing DPO machinery. This suggests a general recipe that could be ported to other modality pairs (video-text, audio-text, sensor-text) and offers a concrete alternative to post hoc decoding-time interventions, which the authors note cannot change a model's internal decision boundaries.

Real-world applications:

• Accessibility tools that describe video content for blind or low-vision users, where falsely "hearing" an event implied by visuals would produce incorrect narration.
• Media monitoring and content indexing systems that must tag what is actually present in audio versus video without cross-contaminating the two streams.
• Interactive assistants and agents that, as the paper frames it, "see and listen" before they think and respond — for instance in robotics or smart-home settings where a spoken command should not be confused with a visual cue.
• Emotion and affect analysis in audiovisual content, an application the paper cites in related work through AVEm-DPO-style modality-specific preferences.

Industry relevance. The method is model-agnostic (demonstrated on Qwen 2.5 Omni at 7B and MiniCPM-O 2.6 at 8B parameters), builds on an open preference-optimization stack (LLaMAFactory), and trains for only one epoch, making it practical for teams that already run DPO pipelines. Because it improves hallucination resistance and general audiovisual, video, and audio benchmark scores, it does not appear to trade off general capability for robustness.

Future Directions

• Promoting beneficial cross-modal synergy. The authors explicitly acknowledge that their framework does not promote positive cross-modal interactions for tasks that genuinely require complementary audio and visual reasoning — for example, confirming a visual entity (a dog) via its sound (barking). A trained objective that encourages useful cross-modal evidence without reintroducing spurious correlations is an open problem.
• Extending the objective to joint audiovisual tasks. The paper includes a joint audiovisual loss in the supplementary material, but experiments showed it delivered no significant gains. Understanding why, and designing a formulation that does help on tasks like audiovisual captioning, remains unresolved.
• Generalizing beyond audio and video. The invariance/sensitivity formulation is written for two modalities; extending it to three or more (for example, adding tactile or sensor streams) is a natural next step.
• Improving corruption strategies. The ablation shows diffusion-based corruption at 500 steps works best and that low-step corruption actively hurts. Better ways to construct corrupted inputs — and to select hard negatives — could tighten the objective further.

Target Audience

This paper is most useful to researchers and engineers working on multimodal or omni-modal LLM alignment, particularly those already familiar with DPO and looking for training-time alternatives to decoding-time hallucination defenses. It will also interest practitioners who need reliable audiovisual understanding in production systems and want a method that fits into existing preference-tuning infrastructure without a separate reward model. Readers with a background in multimodal benchmarks such as AVHBench and CMM will get the most out of the results tables; readers new to preference optimization will need to work through the DPO preliminaries in Section 3 first.

Authors’ abstract

Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.