Hallucination Begins Where Saliency Drops

AI summary

Overview

Research area: Computer Vision / multimodal large language models — specifically the mechanistic analysis and inference-time mitigation of hallucinations in Large Vision-Language Models (LVLMs).

Technical level: Advanced. The paper assumes familiarity with transformer attention, gradients, cross-entropy loss, autoregressive decoding, and saliency-based interpretability.

Scope: The paper introduces a gradient-aware token-level saliency metric (LVLMs-Saliency), establishes a link between low output-token saliency and hallucination, and proposes two inference-time interventions (SGRS and LocoRE) evaluated across eight LVLM configurations and multiple image benchmarks.

What This Paper Is About

LVLMs frequently "hallucinate" — describing objects that are not in the image (e.g., generating "blue" for a wallpaper). Prior work blamed attention sinks and relied only on forward-pass attention maps, which the authors argue cannot reliably separate correct outputs from hallucinated ones. This paper asks not just when hallucination happens but why, and proposes that hallucinations occur when previously generated output tokens exert low influence on the next-token prediction — a breakdown of contextual memory. Using that diagnosis, the authors build two lightweight decoding-time mechanisms to prevent it.

Key Contributions

1. LVLMs-Saliency: An unsupervised, gradient-based metric defined as the element-wise (Hadamard) product of attention weights and their gradients, masked to the lower triangle and L2-normalized across heads. It quantifies how strongly each prior output token influences the next-token prediction, yielding a token-level view of contextual grounding.

2. The "Pattern 1" finding: Systematic analysis establishes that hallucinations occur when prior output tokens show low saliency to the next token prediction, indicating a failure of contextual memory. The authors frame this as a direct causal link between low output-token saliency and hallucination, distinct from the attention-sink explanation.

3. Saliency-Guided Rejection Sampling (SGRS): The first inference-time mechanism that filters candidate tokens by their saliency with respect to prior output context, rejecting candidates below a context-adaptive threshold and resampling before the token is committed to the sequence.

4. Local Coherence Reinforcement (LocoRE): A lightweight, plug-and-play module that strengthens attention from the current query to its most recent w_s output tokens using a distance-aware gain factor. Unlike prior methods that rebalance cross-modal (image) attention, LocoRE operates purely within the output text stream, requiring no gradient computation or parameter modification.

Main Findings

• Attention maps alone are insufficient: In Figure 1 (Qwen2-VL-7B), attention maps for the correct token "wallpaper" and the hallucinated token "blue" look visually similar, while saliency maps show structured grounding for the correct token and collapse for the hallucinated one. Figure 2 shows the same pattern: for correct tokens, saliency to recent outputs is high and typically decays with distance; for hallucinated tokens, saliency toward all prior outputs collapses.

• Prompt saliency is not the main cause: Analysis of 500 samples found that while saliency of user prompts differs between correct and hallucinated tokens, these scores do not significantly affect the model's predictive accuracy.

• Hallucination reduction on LLaVA-1.5-7B (Table 1): LocoRE reaches POPE F1 86.9 and accuracy 87.3, with CHAIR_S 38.4 and CHAIR_I 11.2 (baseline Beam Search: 85.4 / 84.0 / 51.0 / 15.2). SGRS + LocoRE reaches POPE F1 87.0 and accuracy 87.5, with CHAIR_S 35.6 and CHAIR_I 8.2. On MME, SGRS + LocoRE totals 668.33 versus the baseline 565.34.

• Consistent gains across models (Table 2): With SGRS + LocoRE, LLaVA-1.5-7B improves LLaVA W by +4.2, MM-Vet by +5.5, VizWiz by +6.4, SQA by +2.3, CHAIR_S by +12.4 and CHAIR_I by +5.7. LLaVA-1.5-13B improves LLaVA W by +4.3, MM-Vet by +5.9, VizWiz by +3.5, SQA by +3.4, CHAIR_S by +7.4 and CHAIR_I by +4.8. Qwen2-VL-7B improves LLaVA W by +4.1, MM-Vet by +4.5, VizWiz by +3.0, CHAIR_S by +5.7 and CHAIR_I by +2.2. Qwen2.5-VL-7B improves by +3.2, +4.0, +1.8, +6.2 and +2.5 respectively. Intern-VL-7B and Intern-VL-13B also improve on all listed metrics.

• Larger models benefit less: Qwen2.5-VL-32B is only evaluated with LocoRE in Table 2, showing smaller gains (+0.5 LLaVA W, +0.9 MM-Vet, +0.4 VizWiz, +0.3 SQA, +1.8 CHAIR_S, +1.0 CHAIR_I).

• LocoRE restores saliency (Figure 3): Without LocoRE, generating the incorrect token "clock" shows low saliency to prior outputs; with LocoRE, the same position generates the correct token "watch" with significantly higher saliency to recent outputs.

• Hyperparameters (Table 3): With α = 0.6, β = 0.15 on LLaVA-1.5 and α = 0.6, β = 0.20 on Qwen2-VL, CHAIR_S drops to 35.6 and 19.3, and POPE F1/accuracy reach 87.0/87.5 and 88.0/89.0. SGRS alone (α = 0.6, β = 0.0) contributes most of the improvement, with LocoRE adding further gains (LLaVA-1.5 POPE F1 from 85.4 to 86.9). Setting β = 1.0 collapses performance (LLaVA-1.5 CHAIR_S 50.2, POPE F1 60.3).

• Reported hallucination-rate reduction: The full configuration reduces the CHAIR hallucination rate by 28.3% (LLaVA-1.5) and 22.8% (Qwen2-VL) versus baseline.

• Latency cost: SGRS requires a backward pass, adding 30–40% overhead per token versus standard greedy decoding. LocoRE alone, as a forward-only in-place attention modification, incurs <2% latency increase. Raising α to 0.9 lowers CHAIR_S from 35.6% to 30.0% and raises POPE from 87.0% to 87.1% but costs 33% more latency (30.8 ms/token to 41.2 ms/token).

Methodology in Plain English

The authors start from the observation that simply looking at where a model "looks" (its attention weights) does not explain hallucination. Instead they ask: if you nudge a particular attention weight, how much does the model's output change? Multiplying the attention weight by its gradient answers that question — this product is the saliency score. Summing it over all previously generated output tokens tells you how much the model's own earlier words are still shaping the word it is producing now.

Using this lens, they find that when the model produces a correct word it keeps strong ties to its recent output history; when it hallucinates, those ties essentially vanish — the model "forgets" what it just said.

To fix this they add two interventions that run during decoding, without retraining:

1. SGRS samples several candidate next tokens, scores each one's saliency toward prior outputs, and compares it to an adaptive threshold. That threshold is a fraction (α) of the average saliency of the most recent W output tokens — so the bar rises and falls with how well-grounded the recent generation has been. Low-saliency candidates are discarded and resampled (up to R times); if all are rejected, the highest-saliency candidate is taken as a fallback.

2. LocoRE then reinforces the attention links in the next forward pass, multiplying attention from the upcoming query to the last w_s output tokens by a gain of 1 + β. The paper uses Sys_L = 35 and Img_L = 576 to identify where output tokens begin for LLaVA-1.5.

The two form a closed loop: SGRS gates what gets in, LocoRE keeps it from being forgotten. A reformulation in Appendix A expresses saliency via a Taylor expansion as the absolute value of the head-averaged element-wise product of the attention matrix and the loss gradient.

Why This Matters

Impact on research: The paper offers an alternative to the prevailing attention-sink explanation of hallucination, arguing that gradient information is necessary to distinguish hallucinated from correct outputs. It shifts the diagnostic focus from image attention (as in EAH, TAME, Farsight, Vissink) to the dynamic saliency of the model's own output stream, and provides an interpretability tool that is unsupervised and requires no labels.

Real-world applications:

• Medical or industrial image captioning, where a fabricated object or finding carries high cost.
• Assistive technologies for blind and low-vision users, where captions must not invent objects.
• Autonomous systems and robotics, where vision-language descriptions feed downstream decisions.
• Content moderation and search indexing, where unreliable visual descriptions degrade downstream quality.

Industry relevance: The methods are training-free and plug-and-play, and LocoRE in particular needs no auxiliary models, no external detectors, and no multi-pass decoding, at <2% latency overhead — an attractive profile for teams deploying existing LVLMs without retraining budgets. The 30–40% overhead of SGRS, however, makes the full pipeline less suited to real-time serving.

Future Directions

• Reducing SGRS latency: The gradient-based backward pass is the dominant cost (30–40% per token). Approximating saliency without backpropagation would make the full pipeline viable for real-time use.
• Understanding β sensitivity: Table 3 shows β = 1.0 causes a severe collapse in both CHAIR and POPE scores, while the text also recommends β = 1.2 at one point — the safe operating range and the cause of this degradation are not fully explained in the provided content.
• Scaling behavior: Gains shrink on the largest model tested (Qwen2.5-VL-32B). Whether saliency-based intervention is unnecessary for stronger models, or needs recalibration, is an open question.
• Cross-modal extension: LocoRE deliberately operates only within the output text stream. Whether the same saliency signal can diagnose and correct image-token grounding failures — or failures on video and other modalities — is left open.

Target Audience

Researchers and engineers working on multimodal LLM reliability, interpretability, and decoding strategies; practitioners seeking training-free hallucination mitigation they can attach to existing LLaVA, Qwen-VL, or Intern-VL deployments; and mechanistic-interpretability researchers interested in gradient-based alternatives to attention-map analysis. Readers should be comfortable with transformer internals and benchmark metrics such as POPE, CHAIR, MME, MM-Vet, VizWiz, and ScienceQA.

Authors’ abstract

Recent studies have examined attention dynamics in large vision-language models (LVLMs) to detect hallucinations. However, existing approaches remain limited in reliably distinguishing hallucinated from factually grounded outputs, as they rely solely on forward-pass attention patterns and neglect gradient-based signals that reveal how token influence propagates through the network. To bridge this gap, we introduce LVLMs-Saliency, a gradient-aware diagnostic framework that quantifies the visual grounding strength of each output token by fusing attention weights with their input gradients. Our analysis uncovers a decisive pattern: hallucinations frequently arise when preceding output tokens exhibit low saliency toward the prediction of the next token, signaling a breakdown in contextual memory retention. Leveraging this insight, we propose a dual-mechanism inference-time framework to mitigate hallucinations: (1) Saliency-Guided Rejection Sampling (SGRS), which dynamically filters candidate tokens during autoregressive decoding by rejecting those whose saliency falls below a context-adaptive threshold, thereby preventing coherence-breaking tokens from entering the output sequence; and (2) Local Coherence Reinforcement (LocoRE), a lightweight, plug-and-play module that strengthens attention from the current token to its most recent predecessors, actively counteracting the contextual forgetting behavior identified by LVLMs-Saliency. Extensive experiments across multiple LVLMs demonstrate that our method significantly reduces hallucination rates while preserving fluency and task performance, offering a robust and interpretable solution for enhancing model reliability. Code is available at: https://github.com/zhangbaijin/LVLMs-Saliency