Mechanisms of Prompt-Induced Hallucination in Vision-Language Models

AI summary

Overview

Research area: Mechanistic interpretability of vision–language models (VLMs), specifically the study of hallucination caused by misleading textual prompts.

Technical level: Intermediate. Readers benefit from familiarity with transformer attention heads, ablation-style interventions, and multimodal benchmark evaluation.

Scope: The paper isolates, localizes, and suppresses prompt-induced hallucination (PIH) in three 7B-parameter VLMs by ablating a small set of attention heads, and characterizes how those heads mediate prompt copying across counting and color tasks.

What This Paper Is About

Vision–language models sometimes describe things that are not in the image when the text prompt tells them those things are there — for example, being asked to describe four waterlilies when only three are present, and then describing a fourth flower that does not exist. The paper asks when this prompt-induced hallucination happens, where inside the model it is implemented, and whether it can be removed without retraining the model.

Key Contributions

1. Introduces and analyzes PIH as a failure mode. The authors formalize prompt-induced hallucination as a setting where VLMs prioritize prompt information over conflicting visual evidence, and characterize when it occurs as a function of object count and discrepancy size (Section 3).
2. Identifies a small set of causally responsible attention heads. Through mean-ablation knockouts, the authors find per-model sets of "PIH heads" whose ablation substantially reduces PIH across models and tasks, with no additional training and without degrading performance on aligned prompts (Sections 4 and 5).
3. Characterizes the function of PIH heads. Analysis of copying form, attention mass, and output probabilities shows that ablation suppresses prompt copying while increasing correction toward visual evidence — but through model-specific mechanisms (Section 6).

Main Findings

• PIH depends strongly on object count. For images with small numbers of objects (2–4), models typically resist misleading prompts and report the true count, though they still produce PIH in roughly 20–40% of cases, primarily when the discrepancy is small (N+1). Beyond four objects, behavior reverses: prompt-matching responses approach 80–90% and ground-truth responses drop to near zero, even when the prompt substantially overestimates the count.

• Discrepancy size stops mattering at high counts. For images with more than four objects, prompt-aligned response rates converge across different discrepancy distances. The effect persists at extreme offsets (k ∈ {10, 20, 50}), with models describing, for example, fifty-nine cats in an image containing only nine.

• PIH tracks low base confidence. Under the baseline prompt "How many [objects] are there in the image?", probability assigned to the true count N is high for N ≤ 4 and low for N ≥ 5, where probability mass spreads to neighboring counts N+k. The Pearson correlation between base probability and confirmation to the prompt is ρ = 0.37 and 0.46 for Qwen-VL and Janus-Pro respectively, and 0.1 for LLaVA-OneVision.

• Ablating PIH heads sharply reduces prompt-following. On misaligned prompts, prompt-match rates drop from 42–64% before intervention to below 11% for all models after ablation (LLaVA-OneVision 42.58 → 1.42, Qwen-VL 56.51 → 3.22, Janus-Pro 64.10 → 10.19). Random ablation of an equal number of heads from the same layers produces only marginal reductions.

• Correction toward the true count rises correspondingly. True-count match rates reach 70–78% after ablation (LLaVA-OneVision 45.68 → 77.80, Qwen-VL 37.70 → 70.66, Janus-Pro 30.54 → 70.90).

• General counting ability is preserved or improved. Baseline-prompt exact match after ablation is 81.24 (LLaVA-OneVision, +4.35 points), 79.29 (Qwen-VL), and 79.41 (Janus-Pro). The object mentioned in the response remains unchanged and correct.

• Broader capabilities are stable. On CalTech101, MM-Vet, and POPE, accuracy is broadly unchanged after PIH-head ablation (for example LLaVA-OneVision: 100.0/100.0, 50.5/50.7, 89.3/89.3 before/after; Qwen-VL: 100.0/96.0, 43.9/42.8, 85.0/86.0; Janus-Pro: 99.0/99.0, 46.5/44.8, 86.3/87.3).

• PIH heads concentrate in early layers of the language model. Among the top-10 PIH heads, 5/10 occur in layer 0 for Qwen-VL, 7/10 for LLaVA-OneVision, and 3/10 fall within the first two layers for Janus-Pro. The top-ranked heads are L0H3 and L0H6 in both Qwen-VL and LLaVA-OneVision, and L0H20 in Janus-Pro.

• PIH appears to be language-model-internal. Qwen-VL and LLaVA-OneVision share the same underlying Qwen2 language model but different vision backbones and fusion architectures; their top-1 and top-2 heads overlap perfectly, and at m = 10 half of the identified heads are shared. This suggests PIH is mediated by language-model attention heads rather than vision-specific components.

• The same heads generalize to color. Reusing the counting-derived PIH heads on a color task from Visual CounterFact reduces prompt-induced hallucinations by between 40% and 95% (up to 94.25%). On the color task, the PIH rate falls from 99.04 to 4.79 (LLaVA-OneVision), 85.22 to 44.58 (Janus-Pro), and 79.73 to 20.28 (Qwen-VL).

• PIH is associated with copying the prompt's format. Before ablation, format-copying responses account for over 90% of outputs for LLaVA-OneVision and Janus-Pro on the color task. In counting, Janus-Pro outputs a digit in 43.95% of cases, of which 43.45% are incorrect — indicating incorrect responses stem from exact copying of the misaligned prompt.

• Ablation shifts attention toward the image. The layer with the largest intervention-induced change shows image attention mass increases of Δ = 0.121 in LLaVA-OneVision (layer 2), Δ = 0.053 in Qwen-VL (layer 27), and Δ = 0.037 in Janus-Pro (layer 22).

• The mechanism of copying reduction is model-specific. Janus-Pro primarily inhibits format copying when correcting misaligned prompts (reducing format copying by nearly 30%). Qwen-VL instead increases reliance on the prompt's format — P(N_word | N_digit) decreases by 32.6% while P(N_digit | N_digit) increases by 13.6%, with format copying rising from 40.21% to 53.95%. LLaVA-OneVision suppresses all forms of copying, with P(N_word | N_digit) decreasing by roughly 18% and the largest shift in attention mass toward the image (+12%).

Methodology in Plain English

The authors build a controlled counting test from CountBench (491 text-image pairs, expanded to 3,437 image-prompt pairs). For each image with ground-truth count N, they first ask "How many [objects] are there in the image?" and keep only samples the model answers correctly. They then ask the model to "Describe the N+k [objects] in the image," with k ∈ {1, 2, 3, 4, 5} as the discrepancy distance. Because the prompt is grounded with the phrase "in the image," describing N+k objects counts as hallucination. Models are allowed up to 75 tokens of generation, and accuracy is determined by a rule-based extraction of the first non-negated numerical reference.

To find the responsible components, they perform mean-ablation knockouts of individual attention heads — removing token-specific information carried by the head while preserving its overall activation magnitude — and score heads by how often the response switches from the prompted count to the true count. They rank heads and then ablate the top-m heads simultaneously for m ∈ {1, 3, 5, 10}, selecting m = 3 for Qwen-VL and m = 10 for LLaVA-OneVision and Janus-Pro. Randomly selected heads from the same layers serve as a control.

For the generalization test, they reuse the same per-model head sets on a color task derived from the "color" split of Visual CounterFact (493 base-image-label pairs, expanded to 2,465 image-prompt pairs), using "What color is the [object] in the image?" as the baseline and "Describe the C+k [object]" as the misaligned prompt, where k ∈ {1, 2, 3} denotes perceptual distance on the color wheel. To probe function, they analyze copying forms in the generated text, measure shifts in attention mass from assistant tokens to image tokens, and track output probabilities for the count in digit versus word form. The models studied are LLaVA-OneVision-7B (Qwen2 language model, SigLIP vision encoder), Qwen2-VL-7B (Qwen2, DFN-ViT w/ RoPE-2D), and Janus-Pro-7B (DeepSeek-LLM, SigLIP-Large-Patch16-384). All experiments ran on a single NVIDIA RTX 3090 GPU with 24 GB of memory, consuming approximately 200–300 GPU hours, with no training or fine-tuning.

Why This Matters

Impact on research. The work argues that PIH is not a failure of visual perception or numerical reasoning but a localized, causally identifiable routing behavior: a small set of early-layer language-model attention heads propagates prompt information forward. Because the identified heads overlap heavily between two models sharing the same language backbone, the paper suggests these mechanisms live in the language model rather than in vision-specific components — a claim that reframes how multimodal hallucination should be studied. It also demonstrates a training-free intervention, which contrasts with alignment- or fine-tuning-based remedies.

Real-world applications:

• Assistive and accessibility tools that describe images to users, where a user's mistaken phrasing ("describe the five people in this photo") could otherwise cause the system to invent objects.
• Medical or scientific image reporting, where a clinician's or technician's prompt may carry an assumed count or finding that the model then echoes back.
• Document and diagram question answering, where prompts often state quantities that the model should verify rather than adopt.
• Content moderation and image captioning pipelines, where a user-supplied caption or count can bias automated descriptions.

Industry relevance. The intervention requires no additional data, no retraining, and fits within a single 24 GB GPU, making targeted head ablation a plausible lightweight safeguard for deployed multimodal systems. The finding that general benchmarks such as MM-Vet and POPE are not degraded addresses the central practical objection to surgical model edits.

Future Directions

• Scaling beyond medium-scale models. The study covers only ~7B-parameter VLMs; whether the same PIH heads and ablation effects transfer to substantially larger models is untested.
• Explaining the divergent copying mechanisms. Why PIH ablation reduces soft copying while encouraging format copying in Qwen-VL, and produces a strict decrease in format copying with an increase in soft copying in Janus-Pro, is left for future work. The authors suggest architecture, selected-head patterns, or training data as possible explanations.
• Tracing secondary effects. Ablating PIH heads may induce secondary changes in other attention heads or layers, and the paper does not trace how these emerge — though it notes these secondary effects may be more human-interpretable than the original heads.
• Looking beyond attention-based analysis. The authors acknowledge that attention patterns are often not directly interpretable and that their method offers only a partial view of internal computation, leaving room for complementary interpretability approaches.

Target Audience

Mechanistic interpretability researchers and multimodal modeling practitioners; engineers building VLM-based products who need training-free mitigation of prompt-driven errors; and evaluation researchers interested in how benchmark prompts interact with model internals. Readers without background in attention analysis will still follow the high-level findings but may find the ablation methodology and copying-form taxonomy dense.

Authors’ abstract

Large vision-language models (VLMs) are highly capable, yet often hallucinate by favoring textual prompts over visual evidence. We study this failure mode in a controlled object-counting setting, where the prompt overstates the number of objects in the image (e.g., asking a model to describe four waterlilies when only three are present). At low object counts, models often correct the overestimation, but as the number of objects increases, they increasingly conform to the prompt regardless of the discrepancy. Through mechanistic analysis of three VLMs, we identify a small set of attention heads whose ablation substantially reduces prompt-induced hallucinations (PIH) by at least 40% without additional training. Across models, PIH-heads mediate prompt copying in model-specific ways. We characterize these differences and show that PIH ablation increases correction toward visual evidence. Our findings offer insights into the internal mechanisms driving prompt-induced hallucinations, revealing model-specific differences in how these behaviors are implemented.