Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats

AI summary

Overview

Research area: Computer Vision / Vision-Language Models — specifically hallucination mitigation in Large Vision-Language Models (LVLMs) via internal attention-head intervention.

Technical level: Intermediate to Advanced. The paper assumes familiarity with transformer attention heads, multi-head attention, causal pathways, and decoding-time intervention, though its core argument is explained conceptually.

Scope: The paper proposes "AllPath," a unified, training-free intervention framework that identifies and scales attention heads lying on three distinct causal pathways (image-to-output-text, image-to-input-text, and text-to-text) in order to reduce LVLM hallucinations across both discriminative (Yes/No, multiple-choice) and generative (captioning) question–answer alignment formats.

What This Paper Is About

LVLMs frequently hallucinate — producing text that is inconsistent with the input image — which undermines user trust and risks misinformation. The paper argues that existing intervention-based fixes each target only a single causal path (either image-to-output-text, as in PAI, or text-to-text, as in AD-HH), so no single method works consistently across benchmarks with different question formats. The goal is a framework that diagnoses and intervenes on all relevant paths simultaneously, and adapts which paths it uses to the format of the question being asked.

Key Contributions

1. A multi-path hallucination intervention framework. The authors state this is the first multi-path hallucination intervention framework for LVLMs, aligned with the transformer's input–output sequence and causal architecture, covering text-to-text and image-to-text (image-to-input-text and image-to-output-text) pathways, rather than only one path as in prior work.
2. Two new head-identification methods. One probes text-to-text heads using a refined Log Probability Increase (LPI) score comparing hallucinated versus non-hallucinated token sets; the other probes image-to-text heads using the ratio of key-object attention to total image attention. Both operate on local behavior within a layer and require only a single forward pass, unlike zero-out ablations (which need a full forward pass per head per generation step) or training-based classifiers.
3. New empirical findings about LVLM internals. Through counterfactual edits and correlation analysis of attention heads, the paper reports that text-to-text heads correlate strongly with alignment format; that image-to-input-text and image-to-output-text heads differ substantially despite both attending more to image tokens than other heads; and that LVLMs adaptively select combined pathways depending on question type.
4. AllPath, a simple intervention method with consistent gains. Built on the framework, AllPath selects critical heads and rescales them (multiplying by γ⁺ or γ⁻), improving results across POPE, MCQ-POPE, CHAIR, and MME while remaining lightweight.

Main Findings

• Hallucination is not caused by one path. The paper reports that hallucinations in LVLMs arise from the interplay among the image-to-input-text, image-to-output-text, and text-to-text pathways, so intervening on only one is insufficient.
• Text-to-text heads track the answer format. Using rank correlations of importance scores, the correlation between heads identified from POPE COCO adversarial and POPE COCO random reaches 0.8247, and between POPE COCO adversarial and POPE GQA adversarial reaches 0.8358 — both Yes/No formats. Switching to multiple-choice format drops the correlation to only 0.1233, even with the same images and similar questions.
• Image-to-input-text heads differ from image-to-output-text heads. Heads identified separately from the input and output paths are largely uncorrelated, yet all identified image-to-text heads attend to image content more strongly than the average of all heads. The authors conclude that looking only at the output text when mitigating hallucinations may be insufficient.
• Knocking out image attention hurts open-ended tasks far more than Yes/No tasks. On LLaVA-1.5-7B, masking attention from output tokens to image tokens drops POPE accuracy by roughly 2% (e.g., random split 85.1 → 83.2, popular 83.7 → 82.0, adversarial 80.9 → 79.7), but worsens CHAIR considerably (C_S 52.2 → 62.4, C_I 14.6 → 32.4). The paper describes this as about a 10% worsening on CHAIR, indicating different pathways for different formats.
• AllPath leads on all tested benchmarks. On POPE, AllPath scores 87.2/86.0 (Acc/F1) on Random, 86.0/84.9 on Popular, and 82.8/82.1 on Adversarial, versus the vanilla baseline at 85.1/83.7, 83.7/82.4, and 80.9/80.0. The paper reports at least a 2.1% F1 improvement, versus the best baseline improvement of 0.9%.
• Strong gains on multiple-choice and captioning. On MCQ-POPE (Macro F1), AllPath reaches 80.5/80.5, 74.0/74.0, and 69.7/69.7 for Random/Popular/Adversarial, which the paper describes as at least a 5.2% increase. On CHAIR, AllPath attains C_S 26.6 and C_I 7.2, with the paper reporting a 2.6% F1 increase versus the best baseline improvement of 1.9%. The paper also notes VCD performs well on POPE and MCQ-POPE but declines on CHAIR, while AD-HH is strong on CHAIR and MCQ-POPE but nearly matches vanilla on POPE.
• MME hallucination subset improves with POPE-derived heads. AllPath scores 188.3 (±2.89) existence, 126.1 (±0.96) count, 132.2 (±8.39) position, 153.3 (±7.64) color, for a total of 600.0 (±8.66), versus vanilla at 180.0 (±8.66), 113.9 (±10.05), 116.7 (±15.90), 129.4 (±18.28), total 540.0 (±39.69). Results are averaged over 3 test runs.
• Both head sets are needed. Ablation on POPE shows that using T2T heads alone or I2T heads alone each improves over vanilla (85.1/83.7 Random), but the largest gain comes from using all heads (87.2/86.0 Random).
• Low hyperparameter sensitivity. With ξ/ζ/γ⁺/γ⁻ varied, Random-subset accuracy fluctuates only within 85.6–88.3 while the baseline is 85.1. Some alternative settings (e.g., ξ=30) exceed the default, which the authors attribute to choosing a single set of hyperparameters that works across discriminative tasks rather than tuning per dataset.

Methodology in Plain English

The approach has four parts.

1. A three-path mental model. The authors picture an LVLM as having an image that informs the input text, the image that directly informs the output text, and the earlier text that informs the later text. Hallucination can be driven by any of these, so they build a framework covering all of them.

2. Finding "text-to-text" heads. For each attention head, they ask: if this head's output were added on its own, how much would the model's log-probability of a hallucinated token change versus a non-hallucinated token? The difference between those two quantities is the Text-to-Text (T2T) score. A lower value means the head pushes toward hallucination. For short-answer questions, the first decoding step is used, with the correct answer in the non-hallucination set and the wrong alternatives in the hallucination set; for open-ended questions, individual generated tokens judged hallucinated or not are placed in the corresponding sets.

3. Finding "image-to-text" heads. They collect the first mention of each object in the question or the response — first mentions are expected to be grounded in the image rather than inferred from earlier text. Tokens whose objects are in the image should show strong, concentrated attention inside the matching image region; tokens whose objects are absent should show weaker, more diffuse attention. The Image-to-Text (I2T) score is the difference between those two averaged attention quantities.

4. Intervening. Once heads are scored, the top ξ heads by alignment score and the bottom ξ heads are taken as suppressors and promoters respectively, and the top ζ heads by I2T score are taken as visually well-behaved. At inference, head outputs in the desirable set are multiplied by γ⁺ and those in the hallucination-promoting set by γ⁻. Defaults are γ⁺ = 2.0 and γ⁻ = 0.0; ξ = 20 and ζ = 10 for short-answer tasks, and ξ = 40 and ζ = 50 for open-ended tasks. To avoid overfitting to test questions, heads are identified on newly generated data that shares only the format of each benchmark, never its images.

Experimental setup. Evaluations use LLaVA-1.5-7B (with Qwen-VL-Chat and Qwen2.5-VL-7B/72B results placed in the appendix). POPE draws 500 images each from COCO, A-OKVQA, and GQA with three present and three absent objects per image, under random, popular, and adversarial sampling, yielding 3,000 questions per dataset. MCQ-POPE is a new benchmark the authors built by converting POPE into four-option multiple-choice questions, also with 3,000 questions per dataset and split. CHAIR measures hallucinated objects in captions at sentence (C_S) and instance (C_I) level. MME is reported on the hallucination subsets (existence, count, position, color) using the sum of accuracy and accuracy+. Baselines are VCD, ICD, PAI, and AD-HH. POPE and MCQ-POPE use nucleus sampling with temperature 1 and no beam search, with the prompt suffix "Please answer this question with one word." CHAIR samples 500 COCO validation images, uses the prompt "Please help me describe the image in detail," and generates greedily up to 512 tokens.

Why This Matters

Impact on research. The paper reframes hallucination mitigation as a multi-path problem rather than a single-path one, and supplies evidence that head function is tied to answer format (correlation 0.8247–0.8358 within Yes/No formats versus 0.1233 when moving to multiple-choice). This challenges the common practice of evaluating one intervention method on one benchmark and claiming general success, and it introduces two head-scoring methods that need only a single forward pass compared with per-head ablation or classifier training.

Real-world applications:

• Assistive and accessibility tools that describe images to blind or low-vision users, where fabricated objects in a caption are directly harmful.
• Medical or industrial image reporting, where a model asserting the presence of a finding or defect that is not in the image carries real risk.
• Content moderation and fact-checking pipelines that use vision-language models to caption or index media at scale.
• Autonomous and robotic perception, where a language layer that hallucinates objects could mislead downstream decision-making.

Industry relevance. Because AllPath is training-free, operates in a single forward pass, and is reported to be insensitive to hyperparameters (Random accuracy staying within 85.6–88.3 across a wide sweep), it is a plausible drop-in inference-time wrapper for deployed LVLMs, including ones already fine-tuned on proprietary data where retraining is expensive. Its ability to improve both discriminative and generative formats means one intervention configuration can serve a mixed-query production system rather than requiring separate fixes per task.

Future Directions

• Extending beyond the tested models. The main analysis centers on LLaVA-v1.5-7B, with Qwen-VL-Chat and Qwen2.5-VL-7B/72B results deferred to the appendix; how well the pathway structure generalizes across architectures, sizes, and modality connectors is an open question.
• Automatic pathway selection. The paper shows LVLMs adaptively weight pathways by question type, but AllPath's per-format head sets and coefficient choices are still specified by the authors; learning the pathway mixture per query automatically is a natural next step.
• Better hallucination labels for open-ended generation. The T2T score depends on judgments of which generated tokens are hallucinated versus not, and the reliability and cost of that labeling at scale is not resolved here.
• Understanding why input-path and output-path heads diverge. The paper establishes that image-to-input-text and image-to-output-text heads are largely uncorrelated while both attend to the image, but does not explain the mechanism; a causal account could enable more surgical interventions.

Target Audience

This paper is most useful to researchers and engineers working on vision-language model reliability, interpretability, and inference-time hallucination mitigation — particularly those who need a method that does not require retraining. It also suits practitioners deploying LVLMs in captioning or visual question-answering products, who can apply the head-scaling recipe directly, and interpretability researchers interested in how attention heads map onto causal pathways and how that mapping shifts with question format. Readers without a background in transformer attention will need to work through the preliminaries section and the score definitions before the analysis findings are fully accessible.

Authors’ abstract

Despite their impressive performance across a wide range of tasks, Large Vision-Language Models (LVLMs) remain prone to hallucination. In this study, we propose a comprehensive intervention framework aligned with the transformer's causal architecture in LVLMs, integrating the effects of different intervention paths on hallucination. We find that hallucinations in LVLMs do not arise from a single causal path, but rather from the interplay among image-to-input-text, image-to-output-text, and text-to-text pathways. For the first time, we also find that LVLMs rely on different pathways depending on the question-answer alignment format. Building on these insights, we propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored to discriminative and generative formats. Experiments across multiple benchmarks demonstrate that our approach consistently reduces hallucinations across diverse alignment types.