Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs

AI summary

Overview

• Research area: Computer vision and multimodal machine learning, specifically hallucination mitigation in Large Vision-Language Models (LVLMs).
• Technical level: Advanced. The paper builds on structural causal models, do-calculus, mediator interventions, and attention-level manipulation inside transformer decoders.
• Scope: The paper proposes Owl (Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation), a causally-grounded framework that treats decomposed visual and textual attention as causal mediators and uses a new metric, VTACR, to drive token- and layer-wise attention reweighting plus dual-path contrastive decoding.

What This Paper Is About

LVLMs often mention objects that are not present in the image, a failure mode called object hallucination. Existing decoder-based fixes usually adjust either visual attention or textual attention in isolation, ignoring the fact that hallucination arises from an imbalance between the two. This paper models that interaction explicitly with a causal graph, introduces a metric to measure the imbalance during decoding, and intervenes on both modalities together to suppress hallucinated content while keeping outputs informative.

Key Contributions

1. The VTACR metric. A new metric, Visual-to-Textual Attention Contribution Ratio, defined per layer as the ratio of average visual-prefix attention to average textual-prefix attention (query plus history), used to quantify cross-modal reliance for the current token during generation.
2. A structural causal model of hallucination. Visual attention (A_V) and textual attention (A_T) are formalized as mediators between inputs, modality priors, and the output, allowing soft interventions of the form do(A_V = A_V*) and do(A_T = A_T*) without altering the input. Total Causal Effect (TCE) is used as the evaluation metric for these interventions.
3. VTACR-guided adaptive attention modulation. Rather than fixed scaling factors, attention coefficients are adjusted per token and per layer whenever a layer's VTACR falls below a layer-wise baseline score derived from the τ-th percentile of a hallucinated-sample distribution.
4. Dual-path contrastive decoding (DCD). Two decoding paths are constructed — a visual-favored path that amplifies image-token attention and attenuates text-token attention, and a text-favored path that does the reverse — and contrasted in a single softmax fusion to widen the gap between faithful and hallucinated predictions.

Main Findings

• Hallucinated tokens show low VTACR. Analysis with LLaVA-1.5 shows hallucinated tokens tend to exhibit skewed VTACR values, indicating over-reliance on the textual modality and weakened visual grounding, consistently across layers and backbones.
• Modality interventions have opposite effects. Increasing visual attention lowers hallucination scores but shortens outputs; increasing textual attention lengthens outputs but worsens hallucination — establishing both attentions as causal mediators and motivating a balanced rather than separate treatment.
• CHAIR results. Owl achieves the best C_S / C_I on all three backbones: LLaVA-1.5 (26.2 / 8.1, length 98.4), MiniGPT-4 (21.2 / 6.2, length 73.6), and Shikra (29.3 / 9.7, length 108.2). Against the strongest prior method PAI, the paper reports C_S reductions of 17.6%, 14.5%, and 22.1%, and C_I reductions of 21.4%, 36.7%, and 24.8% on LLaVA-1.5, MiniGPT-4, and Shikra respectively, achieved while preserving or improving generation length.
• POPE results. Owl leads on all three splits for LLaVA-1.5 (Random 90.2, Popular 88.1, Adversarial 90.5), MiniGPT-4 (82.2, 78.4, 79.0), and Shikra (85.2, 82.3, 83.4). The paper notes Owl slightly trails PAI on MiniGPT-4 and LLaVA-1.5 under the Popular setting, which it attributes to that setting favoring high-frequency objects and aligning with PAI's textual-inertia behavior.
• Vision-language understanding is preserved. On five VQA benchmarks (VQAv2, GQA, VizWiz, ScienceQA-IMG, TextVQA), Owl matches or exceeds base LVLMs on several tasks. On LLaVA-1.5, TextVQA improves by +3.7 and VizWiz rises from 48.8 to 52.5 (7.6% up), with only a marginal VQAv2 decrease from 80.0 to 78.2 (2.3% down). On MiniGPT-4, GQA improves from 44.5 to 45.8 (+1.3) while VQA-v2 dips from 45.0 to 43.2 (−1.8).
• GPT-4V assisted evaluation. Correctness on LLaVA-1.5 rises from 5.58 to 6.70 (20.1%) and Detailedness from 5.30 to 5.90 (11.3%). MiniGPT-4 gains 11.3% in Correctness (5.75 to 6.40) and 5.5% in Detailedness (5.50 to 5.80). Shikra gains 9.1% in Correctness (5.50 to 6.00) and 7.8% in Detailedness (5.10 to 5.50).
• Hyperparameters play complementary roles. Increasing α strengthens visual grounding and reduces hallucination but suppresses informative content and lowers F1; increasing β steadily reduces hallucination with minimal F1 drop; λ between 0.1 and 0.4 gives stable improvements while excessively high values harm both CHAIR and F1.
• Headline claim. The conclusion states a 22.9% reduction in hallucination rates over strong baselines, and the contributions list a 22.9% improvement on CHAIR.

Methodology in Plain English

The authors start by splitting the decoder's attention into two measurable quantities: how much attention the current token pays to the visual prefix, and how much it pays to the textual prefix (the instruction and the generation history). Dividing the first by the second gives VTACR for each layer. They then draw a causal diagram in which image input and text input, along with visual and language priors, feed into visual attention and textual attention, which in turn produce the output. Since the priors cannot be observed or manipulated directly, the authors intervene on the attention values themselves — the mediators — which leaves the inputs unchanged and makes the manipulation interpretable.

To calibrate the intervention, they sample 2,000 hallucinated examples from MSCOCO, compute VTACR for hallucinated tokens in every decoder layer, and estimate a per-layer density distribution. The τ-th percentile of that distribution becomes a per-layer baseline. During generation, if the current layer's VTACR falls below its baseline, the model is judged to be under-grounding and the visual and textual attention coefficients are increased; otherwise they are left untouched. This makes the correction token-by-token and layer-by-layer rather than uniform.

Finally, the model is run twice with opposing biases. One run boosts attention to image tokens and dampens attention to text tokens; the other does the opposite. The two resulting logit distributions are combined in a single softmax with a contrast strength λ, so that predictions supported by visual evidence are amplified and predictions driven by textual priors are pushed down. The authors describe the overall loop as measure, adjust, and decode.

Why This Matters

• Research impact. The paper is one of the few to treat visual and textual attention jointly as causal mediators in LVLM hallucination rather than intervening on one modality or on inputs, and it contributes VTACR as a diagnostic signal that could be reused by other decoding-time methods.
• Medical imaging. The introduction names medical imaging as a safety-critical domain where fabricated object mentions undermine trust.
• Robotic navigation. The introduction also names robotic navigation as a domain where hallucinated content poses serious risk.
• Accessibility tools. The VizWiz gain (48.8 to 52.5) is directly relevant to assistive captioning for users who photograph visually degraded or text-heavy content.
• Image captioning and content generation. The CHAIR evaluation targets open-ended captioning, where hallucinated object names propagate into downstream search, indexing, or moderation pipelines.
• Industry relevance. Owl is a decoding-time intervention applied to existing checkpoints (LLaVA-1.5, MiniGPT-4, Shikra) rather than a retraining procedure, so it does not require the costly human-preference annotation that the paper identifies as a limitation of alignment-based approaches. All experiments ran on 4× NVIDIA 3090 GPUs, indicating commodity hardware is sufficient for adaptation.

Future Directions

• Whether the method generalizes beyond object hallucination. The paper only measures object-level hallucination through POPE, CHAIR, and GPT-4V judging; attribute, relation, and counting hallucination are not evaluated.
• Extending mediator interventions beyond the studied backbones. Only LLaVA-1.5, MiniGPT-4, and Shikra are tested; broader architecture families and larger models are left open.
• Reducing hyperparameter tuning effort. α, β, and λ are empirically tuned per model — (0.4, 0.5) for LLaVA-1.5, (0.2, 0.3) for MiniGPT-4, (0.5, 0.3) for Shikra, λ fixed at 0.2, T at 0.2, τ at 80 — which the ablation shows must be balanced against F1.
• Causal control as a general tool. The conclusion explicitly frames the work as opening avenues for causal control in multimodal generation, inviting follow-up work that applies mediator-level intervention to other generation failures.

Target Audience

Researchers and graduate students working on vision-language models, multimodal decoding strategies, and mechanistic interpretability; practitioners who need to deploy LVLM captioning or VQA systems where fabricated object mentions carry real cost; and readers with a background in causal inference who want to see do-calculus and mediator analysis applied to transformer attention. Readers without familiarity with transformer attention internals or causal graphs will find the methodology section demanding.

Authors’ abstract

Object hallucination remains a critical challenge in Large Vision-Language Models (LVLMs), where models generate content inconsistent with visual inputs. Existing language-decoder based mitigation approaches often regulate visual or textual attention independently, overlooking their interaction as two key causal factors. To address this, we propose Owl (Bi-mOdal attention reWeighting for Layer-wise hallucination mitigation), a causally-grounded framework that models hallucination process via a structural causal graph, treating decomposed visual and textual attentions as mediators. We introduce VTACR (Visual-to-Textual Attention Contribution Ratio), a novel metric that quantifies the modality contribution imbalance during decoding. Our analysis reveals that hallucinations frequently occur in low-VTACR scenarios, where textual priors dominate and visual grounding is weakened. To mitigate this, we design a fine-grained attention intervention mechanism that dynamically adjusts token- and layer-wise attention guided by VTACR signals. Finally, we propose a dual-path contrastive decoding strategy: one path emphasizes visually grounded predictions, while the other amplifies hallucinated ones -- letting visual truth shine and hallucination collapse. Experimental results on the POPE and CHAIR benchmarks show that Owl achieves significant hallucination reduction, setting a new SOTA in faithfulness while preserving vision-language understanding capability. Our code is available at https://github.com/CikZ2023/OWL