SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense

AI summary

Overview

Research area: Computer vision and multimodal learning — specifically object hallucination in Large Vision-Language Models (LVLMs), approached from the visual encoder rather than the language model.

Technical level: Intermediate. Readers need familiarity with CLIP-style visual encoders, vision-language architectures (LLaVA, InstructBLIP, Qwen-VL), and decoding strategies such as contrastive decoding.

Scope: The paper diagnoses three encoder-side causes of object hallucination and proposes SHIELD, a training-free inference framework that combines token re-weighting, noise-derived token subtraction, and adversarial contrastive decoding.

What This Paper Is About

LVLMs frequently describe objects that are not in the image, and prior mitigation work has almost entirely targeted the language-model side of the pipeline. This paper traces the problem to the visual encoder and argues that three distinct defects there — statistical bias, inherent bias, and vulnerability to perturbation — drive hallucination. The goal is to correct those defects at inference time, without any retraining, gradient updates to model weights, or auxiliary trained modules.

Key Contributions

1. Encoder-centric diagnosis. The paper is described as the first to trace LVLM object hallucinations to visual encoders, identifying three issues: statistical bias (overemphasis on frequent visual patterns), inherent bias (erroneous representations of dominant pretraining objects regardless of input), and vulnerability (sensitivity to minor perturbations).
2. SHIELD, a training-free framework. Three combined strategies — token re-weighting to reduce statistical bias, token subtraction using noise-derived tokens to counter inherent bias, and adversarial attack plus contrastive decoding to address vulnerability.
3. Cross-model, cross-benchmark validation. Experiments on LLaVA-1.5, InstructBLIP, and Qwen-VL across CHAIR, POPE, the MME hallucination subset, GPT-4o-assisted LLaVA-Bench evaluation, and AMBER.
4. Evidence beyond hallucination. Improvements on the full MME benchmark indicate the method also lifts general perception, not only hallucination-related scores.

Main Findings

• CHAIR (500 randomly selected COCO 2014 validation images, captions truncated to 512 tokens; lower is better). LLaVA-1.5: SHIELD reaches C_S 36.6 / C_I 10.3, versus 48.8 / 14.2 (vanilla), 46.8 / 13.2 (VCD), and 44.6 / 12.8 (OPERA). InstructBLIP: 40.4 / 10.9 versus 54.6 / 24.8 (vanilla), 44.0 / 13.6 (VCD), 46.4 / 14.2 (OPERA). Qwen-VL: 28.9 / 9.2 versus 49.2 / 13.1 (vanilla), 46.4 / 11.9 (VCD), 34.6 / 9.5 (OPERA). The paper states this is up to 18% improvement over the second-best method, OPERA, on LLaVA-1.5.
• GPT-4o-assisted LLaVA-Bench scores (0–10, higher better). LLaVA-1.5 correctness/detailedness: 6.2 / 6.1 versus 4.9 / 5.0 (vanilla), 5.5 / 5.5 (VCD), 5.6 / 6.0 (OPERA). InstructBLIP: 5.6 / 5.3 versus 4.2 / 4.2 (vanilla), 5.1 / 5.5 (VCD), 5.3 / 5.2 (OPERA). Qwen-VL: 6.9 / 5.8 versus 6.2 / 4.6 (vanilla), 6.5 / 5.7 (VCD), 6.5 / 5.6 (OPERA). Detailedness gains are described as modest, since the method does not target descriptive coverage.
• POPE on the COCO subset (accuracy / F1, higher better). LLaVA-1.5 averages 87.0 / 87.4 versus 81.3 / 79.6 (vanilla), 84.6 / 84.4 (VCD), 84.7 / 85.4 (OPERA). On the hardest Adversarial split, LLaVA-1.5 reaches 82.5 / 83.6 versus 78.9 / 77.5 (vanilla). Qwen-VL averages 87.0 / 86.6 versus 83.6 / 81.6 (vanilla), 86.6 / 86.0 (VCD), 85.2 / 83.3 (OPERA).
• InstructBLIP gains are limited. The paper attributes this to InstructBLIP's Q-Former module constraining the use of modified visual features; its POPE average is 85.0 / 84.8 versus 84.6 / 85.1 for OPERA.
• MME hallucination subset (total scores). LLaVA-1.5: 668.3 versus 565.3 (vanilla), 604.6 (VCD), 592.3 (OPERA), including attribute-level gains in position (148.3 versus 114.0 vanilla) and color (183.3 versus 151.0). InstructBLIP: 461.6 versus 380.3 (vanilla). Qwen-VL: 668.3 versus 587.3 (vanilla).
• AMBER (LLaVA-1.5 7B). SHIELD records the highest AMBER Score of 88.0, versus 82.0 (vanilla), 82.9 (VCD), and 86.5 (OPERA), and outperforms baselines on most metrics across generative and discriminative tasks.
• Full MME (LLaVA-1.5 7B). Total score 1810.8, versus 1632.1 (vanilla), 1717.1 (VCD), 1717.2 (OPERA); perception 1473.0 and cognition 337.8.
• Ablation on CHAIR (C_S, lower better). Vanilla 48.8 → 50.2 with only the adaptive plausibility constraint → 46.4 with the vulnerability module → 40.4 after adding statistical bias mitigation → 36.6 after adding inherent bias reduction. The paper notes the statistical bias step adds a further 13% reduction after vulnerability handling.
• Efficiency (LLaVA-1.5 7B, CHAIR). 7.34 seconds per sample and 18.17 GB peak memory, versus 2.59 s / 15.69 GB (vanilla), 4.89 s / 16.52 GB (VCD), and 24.01 s / 34.88 GB (OPERA). Caption generation alone costs 2.05 s and 16.56 GB, while the inherent-bias module adds only 0.04 s and 0.81 GB.
• Diagnostic statistics in Figure 2. Hallucination proportion rises with stronger token overemphasis measured by peak-to-average L2 norm ratio; with meaningless random-noise images, LLaVA-1.5 frequently hallucinates dominant objects such as cars, chairs, and tables; and F1 on the POPE COCO subset drops sharply after only a few attack steps.
• Attention maps. Visualizations show re-weighting distributes attention across more object-relevant tokens, token subtraction reduces hallucination of dominant pretraining objects, and contrastive decoding reduces the performance drop under increasing attack steps.

Methodology in Plain English

SHIELD adds three inference-time steps that leave model weights untouched.

Token re-weighting. The vanilla model first captions the image with the prompt "Please describe this image." That caption is encoded by the CLIP text encoder paired with the visual encoder, and a normalized similarity matrix is computed between visual tokens and caption tokens. Each visual token's weight is the maximum similarity along the caption dimension, normalized to [0, 1], and the weights are applied as a residual addition to the visual tokens. Hallucinated words in the naive caption do not match visual tokens with high similarity, so the paper argues they do not distort the re-weighting.

Token subtraction. Inherent bias is treated as a property of the encoder parameters, so it can be estimated by feeding K random noise images (the same size as the input) through the encoder and averaging the resulting tokens. That average is subtracted from the re-weighted tokens, producing bias-reduced visual tokens that are fed to the LLM. Because the estimate depends only on the encoder, it can be pre-computed per model.

Adversarial attack plus contrastive decoding. A learnable perturbation is added to the image and refined by gradient descent to minimize the cosine similarity between the perturbed image's global embedding and the caption embedding — an attack tensor that exposes features the encoder handles unreliably. During decoding, the logits from the bias-reduced tokens and the logits from the attacked image are combined contrastively, and an adaptive plausibility constraint zeroes out any token whose probability falls below a fraction of the maximum, following VCD.

Setup. Default hyperparameters are α = 2 (contrastive strength), β = 0.35 (truncation threshold), K = 32 (noise inputs), and l = 0.02 (attack learning rate). Vanilla models and VCD use sampling-based decoding; OPERA uses beam search with a penalty term; SHIELD uses sampling from the modified post-softmax distribution. All experiments run on a single RTX A6000 GPU.

Why This Matters

Research impact. The paper reframes hallucination as an encoder pathology rather than a language-model pathology, complementing a literature (VCD, OPERA, HALC, MARINE, VTI, CLIP-DPO, LURE, LLaVA-RLHF) that has largely focused on the LLM or on training-time fixes. It also supplies diagnostic measurements — the peak-to-average L2 ratio, response to noise-only images, and attack-step degradation curves — that other groups can reuse. A reference implementation is released at https://github.com/hukcc/SHIELD.

Real-world applications.

• Healthcare, where the paper notes hallucination poses reliability and safety risks.
• Autonomous systems, where misdescribed scenes can lead to unsafe decisions.
• Robotics, where language-grounded perception drives action.
• Any deployment of open LVLMs for image captioning, visual question answering, or accessibility descriptions, where false object claims are the primary failure mode.

Industry relevance. Because SHIELD is training-free, it can be layered onto existing deployed LVLMs without labeled data, preference tuning, or RLHF. Its measured overhead (7.34 s per sample, 18.17 GB) is markedly cheaper than OPERA on the same benchmark (24.01 s, 34.88 GB), and the paper notes the trade-off can be tuned through caption length or the number of adversarial optimization steps.

Future Directions

• Designing intrinsically more robust visual encoders to reduce bias and vulnerability at the source rather than patching them at inference.
• Optimizing the adversarial attack mechanism to lower computational cost for real-time applications, which the authors identify as the dominant source of overhead.
• Addressing architectures where modified visual features are bottlenecked, as with InstructBLIP's Q-Former, where SHIELD's gains were

Authors’ abstract

Large Vision-Language Models (LVLMs) excel in diverse cross-modal tasks. However, object hallucination, where models produce plausible but inaccurate object descriptions, remains a significant challenge. In contrast to previous work focusing on LLM components, this paper is the first to trace LVLM hallucinations to visual encoders and identifies three key issues: statistical bias, inherent bias, and vulnerability. To address these challenges, we propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability. Experiments demonstrate that SHIELD effectively mitigates object hallucinations across diverse benchmarks and LVLM families. Moreover, SHIELD achieves strong performance on the general LVLM benchmark, highlighting its broad applicability. Code is available at https://github.com/hukcc/SHIELD.