Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMs

AI summary

Overview

Research area: Multimodal AI — specifically hallucination mitigation in large vision–language models (LVLMs) via inference-time intervention on self-attention.

Technical level: Intermediate. Readers should know what transformers, self-attention, logits, and contrastive decoding are, but the method itself is explained with a short algorithm and needs no specialized mathematics beyond a vector projection.

Scope (one sentence): The paper proposes Attention-space Contrastive Guidance (ACG), a training-free, single-pass method that contrasts an image-conditioned attention output against a mask-approximated text-only attention output inside each self-attention layer, with an orthogonalization step to remove text-aligned bias, and evaluates it on CHAIR, POPE, MMHal-Bench, MMMU, and MathVista across LLaVA-1.5, MiniGPT-4, Qwen-VL, and LLaVA-NeXT 7B/13B.

What This Paper Is About

LVLMs frequently hallucinate because language priors learned from text pre-training override the actual image, producing confident descriptions of objects that are not present. Existing inference-time fixes either correct only the final output logits (requiring multiple forward passes and therefore doubling latency) or apply fixed, heuristic interventions on hidden states or attention maps. This paper instead builds the correction directly into the attention computation: it contrasts what a token attends to with the image against what it would attend to without the image, all within a single forward pass.

Key Contributions

1. Contrastive guidance formulated in attention space. The authors instantiate hallucination mitigation as a contrast between a conditional attention output (attending to all keys) and an approximate text-only (unconditional) attention output, applied at each decoding step to the last text token across self-attention layers. This yields a token-dependent steering direction rather than a single global output-layer correction.

2. A single-pass, masking-based surrogate for the text-only path. Instead of a separate image-absent forward pass, ACG reuses the same score matrix and sets the attention score to negative infinity for visual keys from the last text query, so the unconditional path costs no extra pass.

3. Textual orthogonalization to correct approximation bias. Because the masked surrogate is not a true image-absent state (the paper identifies contextual leakage and softmax redistribution as the two sources of mismatch), ACG projects the guidance vector onto the subspace orthogonal to the normalized unconditional output, removing the component aligned with the text-only direction before applying the guidance scale.

4. Demonstrated faithfulness gains at low cost. ACG improves over training-free baselines on POPE and CHAIR while remaining single-pass, with reported latency reductions of up to 2x relative to multi-pass contrastive decoding, and it generalizes to newer and larger LVLMs (LLaVA-NeXT 7B and 13B).

Main Findings

• POPE overall accuracy. On POPE, ACG achieves the best overall average among compared methods: 86.03 for LLaVA-1.5 (vs. 84.83 Regular, 85.38 VCD, 84.91 PAI, 83.03 VISTA), 76.70 for MiniGPT-4 (vs. 76.31 Regular, 76.59 PAI, 76.19 VISTA), and 86.98 for Qwen-VL (vs. 85.51 Regular, 86.77 VCD). The paper highlights the Adversarial split of LLaVA-1.5 (83.23) and MiniGPT-4 (71.90) as where the improvement is most notable.

• CHAIR hallucination reduction. At max tokens 128 on LLaVA-1.5, ACG reports CHAIR_s 21 and CHAIR_i 4.8 with F1 74.4, against Regular (56.2 / 18.3 / 70.6), VCD (55.0 / 17.0 / 72.5), PAI (25.6 / 7.6 / 75.9) and VISTA (31.0 / 10.5 / 76.6). On MiniGPT-4, ACG reports CHAIR_s 10.8 and CHAIR_i 3.3 at 128 tokens with F1 68.0. ACG is reported as achieving the lowest CHAIR_i across both models and both length budgets (64 and 128).

• Attribute-level and general benchmarks. On MMHal-Bench with LLaVA-1.5, all three reported columns improve (1.94 → 2.12, 37.54 → 38.49, 3.77 → 9.43 at γ = 2). MMMU improves from 35.56 to 36.78 and MathVista from 22.6 to 23.7.

• Generalization to newer/larger models. Under a longer decoding budget of max_new_tokens = 512 with γ = 2.0, ACG on LLaVA-NeXT 7B reduces CHAIR_s from 31.2 to 25.2 and CHAIR_i from 8.1 to 5.4 while raising F1 from 72.1 to 73.9; on LLaVA-NeXT 13B it reduces CHAIR_i from 8.3 to 5.5 and raises F1 from 71.5 to 74.9, while also lowering CHAIR_s from 33.8 to 31.0.

• Orthogonalization helps at matched fidelity (ablation). At similar object-level fidelity, ACG with orthogonalization yields lower hallucination than without it: at roughly 74 F1, ACG (w/ Ortho, γ = 2.4) reports CHAIR_s 21.0 and CHAIR_i 4.8 versus ACG (w/o Ortho, γ = 1.3) at 30.4 and 8.8 — described as 1.8x lower CHAIR_i and 1.4x lower CHAIR_s. At ≈77.6 F1, the with-ortho variant (γ = 2.1) reports 34.2 / 7.6 versus 38.8 / 9.7 without ortho (γ = 1.2).

• Masked unconditional path is empirically justified. Degrading images with Gaussian noise (noise step in {0, …, 999}) raises CHAIR_i sharply from 12.1 to 33.1 and drops F1 from 77.6 to 30.5, with a knee near the 600-step mark, while the mean text-to-image attention ratio shows an overall downward trend, from 9.82% to a minimum of 9.56% at the 700-step mark.

• Guidance scale trade-off. Sweeping γ in [1.0, 3.0] on CHAIR (max 128) with LLaVA-1.5, CHAIR_i decreases from 12.8 at γ = 1.0 to around 5 near γ = 2.4, while F1 stays in the high 70s up to that range; beyond γ ≈ 2.4, F1 drops sharply and captions become overly short. γ = 2.4 is therefore the canonical operating point.

• Earlier layers matter most. In a 32-layer block-wise study, guidance on All layers (γ = 2.4) gives CHAIR_i 4.8, F1 74.4, length 72.4; Early layers (1–8, γ = 2.5) already give substantial reduction (CHAIR_i 7.1, F1 77.5), whereas Early–Mid, Mid–Late, and Late blocks require much larger γ and still yield weaker gains.

• Efficiency. On CHAIR (max new tokens = 128, greedy decoding), Regular latency is 2.81 s per image (1.00x) with CHAIR_i 18.3; VCD is 5.54 s (1.97x) with 17.0; PAI 6.42 s (2.28x) with 7.6; VISTA 5.55 s (1.98x) with 10.5. ACG-Full is single-pass at 3.34 s (1.19x) with CHAIR_i 4.8, and ACG-Fast is single-pass at 2.96 s (1.05x) with CHAIR_i 7.3.

• Qualitative behavior. In MMHal-Bench examples, the baseline LLaVA-1.5 hallucinates in later decoding stages while ACG instead mentions the "brick floor," and in a second example ACG correctly identifies that "This photo is taken at a beach" where the baseline gives an inaccurate scene interpretation.

Methodology in Plain English

The authors treat hallucination as a failure of conditioning: the model drifts toward what language alone would predict instead of what the image shows. Their fix is to compare two versions of what the current token "looks at" inside attention — one that sees the image plus all text, and one where the image has been made invisible — and then push the output away from the image-free version.

The practical obstacle is that a real image-free pass costs a second forward pass, which roughly doubles latency. ACG avoids this by computing the query, key, and value matrices once per layer and then re-running only the softmax with a mask that sets the score to negative infinity for visual keys at the last text token. Both paths therefore share the same computational graph and intermediate states.

Because this mask is only an approximation of a truly image-absent run — earlier layers may already have injected visual content, and masking redistributes attention mass onto text tokens — the naive difference between the two outputs can carry text-induced distortion. ACG corrects this by normalizing the unconditional output into a unit direction and subtracting the component of the difference vector that lies along it, leaving only the part orthogonal to the text-only direction. That corrected vector is scaled by a guidance weight γ and added back to the conditional attention output.

The guided attention output then flows through the usual output projection, residual connection, normalization, and feed-forward network. The one tunable parameter is γ; the paper uses γ = 2.4 for LLaVA-1.5, γ = 0.3 for MiniGPT-4, γ = 1.4 for Qwen-VL, and γ = 2.0 for MMHal, MMMU, and MathVista on LLaVA-1.5 and for both LLaVA-NeXT models. Greedy decoding is used throughout.

Models evaluated: LLaVA-1.5 (CLIP ViT-L/336px encoder, Vicuna language model, two-layer MLP projector, 576 visual tokens), MiniGPT-4 (EVA-CLIP ViT-G/14 with a Q-Former using 32 learnable queries, single linear projection, Vicuna), and Qwen-VL-Chat (ViT-bigG from OpenCLIP, position-aware single-layer cross-attention adapter with 256 queries, Qwen language model). Baselines are VCD (logit-level contrastive decoding), PAI (logit-level classifier-free guidance with attention intervention), and VISTA (latent steering), evaluated only on LVLMs officially supported by the authors' public implementations.

Why This Matters

Impact on research. The paper reframes hallucination mitigation as an objective-driven contrast at the attention layer rather than a post-hoc fix on logits or a hand-crafted attention heuristic. It connects to prior observations of "text inertia," fading visual attention, "visual attention sinks," and

Authors’ abstract

Hallucinations in large vision--language models (LVLMs) often arise when language priors dominate over visual evidence, leading to object misidentification and visually inconsistent descriptions. We address this problem by framing hallucination mitigation as contrastive guidance that steers generation toward visually grounded and semantically faithful text. We propose Attention-space Contrastive Guidance (ACG), a training-free, single-pass method that operates directly in self-attention layers, where hallucination-inducing cross-modal biases emerge. ACG constructs both image-conditioned and approximate text-only attention paths within a single forward pass, enabling efficient guidance before errors accumulate at the output layer. Because this masking-based surrogate can introduce approximation bias, we further apply a lightweight orthogonal projection that suppresses components aligned with the text-only path, yielding a more visually grounded correction. Experiments on CHAIR and POPE show that ACG improves faithfulness over existing training-free baselines while maintaining caption quality, reducing latency by up to $2\times$ compared to multi-pass contrastive decoding methods.