RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models

AI summary

Overview

Research area: Computer Vision / multimodal large vision-language model (LVLM) inference — specifically, decoding-time mitigation of visual hallucinations.

Technical level: Advanced. The paper assumes familiarity with transformer attention, autoregressive decoding, token pruning, and logit-level interventions, and it defines its methods through equations over attention weights and hidden states.

Scope: A training-free, plug-and-play decoding framework (RVSD) that combines semantics-directed visual token sparsification with on-demand retrieval of the discarded tokens to reduce visual hallucinations in LVLMs.

What This Paper Is About

Large vision-language models often generate text that contradicts or is unsupported by the input image, a failure mode called visual hallucination. Prior fixes require curated datasets, extra training, external retrieval databases, or multi-round decoding, all of which add cost. This paper introduces a single-pass decoding method that keeps only the visual tokens relevant to the current generation, stores the rest in a "deferred memory bank," and retrieves from that bank only when the model's own predictive uncertainty signals that grounding has become insufficient.

Key Contributions

1. RVSD framework: A training-free, plug-and-play decoding framework that jointly performs token sparsification and dynamic visual compensation inside a single inference pass, targeting what the authors name the "sparsification-hallucination paradox" — the observation that state-of-the-art sparse decoders' hallucination rates grow substantially as generation length grows (illustrated with VASparse on LLaVA-1.5 using the CHAIR dataset).

2. Semantics-directed token selection: A strategy that scores visual tokens by aggregating cross-modal attention from salient textual positions (weighted by last-layer hidden-state norms), then temporally smooths those scores, unlike prior modality-agnostic pruning criteria designed mainly for acceleration.

3. Semantic-Space Visual Retrieval (SSVR): A mechanism that treats the pruned tokens as a deferred visual memory bank and retrieves visual evidence from it via a query from the current decoder state, triggered by an entropy-based uncertainty gate, then injects the retrieved evidence through a transient adapter that is released after each retrieval step.

4. Empirical validation: Experiments on five benchmarks and three 7B backbones showing state-of-the-art hallucination mitigation, improved decoding efficiency, and robustness under long-context generation.

Main Findings

• POPE (discrimination): RVSD achieves the best accuracy on every split across all three backbones. On LLaVA-1.5-7B the average Accuracy/F1 improves from 80.5/81.6 to 86.2/85.0; on LLaVA-NEXT from 82.4/80.6 to 87.6/86.6; on Qwen-VL from 83.7/81.7 to 85.3/83.5. The largest gains are on the adversarial split: +8.9/+5.4 (LLaVA-1.5), +7.0/+7.3 (LLaVA-NEXT), and +2.4/+2.4 (Qwen-VL). VCD and M3ID improve only marginally, and VTI and AvisC occasionally degrade performance.

• MM-Vet (general capability): RVSD raises the total score from 30.0 to 33.1 on LLaVA-1.5, from 42.1 to 42.7 on LLaVA-NEXT, and from 36.7 to 38.4 on Qwen-VL, with gains on every reported dimension (Rec, OCR, Know, Gen, Spat). Improvements are most pronounced on Know.

• MME hallucination subset: RVSD reaches the highest total on all three backbones — 651.7 (LLaVA-1.5), 668.3 (LLaVA-NEXT), and 593.4 (Qwen-VL). On the two LLaVA backbones the gains concentrate on attribute-level probes (Color and Position), while on Qwen-VL the gain is driven by object Existence, with Count and Position remaining below the strongest baselines.

• CHAIR and AMBER (generative): RVSD attains the best discrimination performance and highest AMBER score on all three backbones, the lowest CHAIR_I on the two LLaVA backbones, and the lowest CHAIR_S on Qwen-VL, indicating suppression of object-level hallucinations during long-form generation.

• Ablation (full MME, LLaVA-1.5): Removing SSVR causes the largest drop, reducing Cognition from 371.79 to 348.21; removing sparse selection slightly raises perception (1514.60 vs. 1511.40) but reduces Cognition by 14.29. Full RVSD scores 1511.40 Perception, 371.79 Cognition, and 1883.19 Total.

• Efficiency: RVSD is the only compared method that keeps latency equal to the Vanilla baseline (1.00x) while reducing memory usage by 2.2%, TFLOPs by 15.6%, and Time-to-First-Token by 20.2%. VCD and M3ID introduce over 2x latency overhead; AvisC incurs the highest cost; VTI increases latency by 1.09x.

Methodology in Plain English

The authors keep the vision-language model frozen and only change what happens during generation.

First, at each decoding step they ask which visual tokens matter for what the model is currently saying. They measure textual "saliency" using the magnitude of the last-layer hidden states, pick the top textual tokens, and aggregate the attention those tokens pay to each visual token across a subset of stable layers. To prevent the selection from jumping around, they exponentially smooth these relevance scores across steps. The top-k scoring visual tokens form an "active sparse set" used for normal decoding; everything else goes into a "deferred set" that becomes a memory bank. This reduces per-layer attention complexity from O((t+N_v)²) to O((t+k)²).

Second, they watch for signs of hallucination. They compute predictive entropy from intermediate layers; when it exceeds a threshold γ, retrieval fires once for that step. The current hidden state serves as a query against the deferred visual tokens using dot-product similarity in a shared semantic space, and the top-k_c candidates are weighted and aggregated. Because the model prunes tokens that look statistically redundant but can be semantically critical, this recovery step is the key repair mechanism.

Third, the retrieved visual evidence is injected through a lightweight transient adapter whose weights are derived from the retrieved features and scale-matched to the host feed-forward network. The result is blended with the residual stream using an injection ratio α, keeping the model's intrinsic distribution mostly intact. The adapter is then released and the trigger reset, so retrieval is parameter-free and paid for only when needed.

Why This Matters

Impact on research. The paper reframes token pruning from an irreversible operation into a reversible, queryable one, and names a concrete failure mode — the sparsification-hallucination paradox — that prior sparse decoders had not systematically characterized. It also shows hallucination mitigation does not have to trade off against efficiency or general multimodal ability, since MM-Vet scores improve alongside hallucination metrics.

Real-world applications.

• Clinical or medical image reporting, where unsupported descriptions of imaging findings carry direct risk (one author is affiliated with a hospital).
• Assistive image description for blind and low-vision users, where fabricated objects mislead the reader.
• Autonomous systems and robotics, where a scene description that contradicts the actual scene is unsafe.
• Content moderation, cataloging, and accessibility captioning at scale, where per-image performance and compute cost both matter.

Industry relevance. RVSD requires no retraining, no annotated data, no external image database, and no detector or draft decoding pass, so it can be dropped into an existing deployed LVLM. The reported efficiency profile — unchanged latency, 2.2% lower memory, 15.6% fewer TFLOPs, and 20.2% lower TTFT — makes it attractive for serving systems where contrastive decoding methods like VCD and M3ID more than double latency.

Future Directions

• Adaptive retrieval gating: The entropy threshold γ is predefined; the authors note that a gate adjusting to input complexity could cut unnecessary retrieval calls on simpler queries, as flagged in their limitations section.
• Stronger retrieval than dot product: The current implementation uses simple dot-product similarity over the deferred memory bank; more advanced retrieval mechanisms are explicitly identified as future work.
• Long-form video understanding: The authors state they will extend RVSD to video, where extended temporal dependencies and richer modality interactions are expected to amplify the sparsification-hallucination paradox.
• Broader model coverage: RVSD is described as model-agnostic, but evaluation is limited to three 7B backbones (LLaVA-1.5, LLaVA-NEXT, Qwen-VL); scaling to larger and structurally different LVLMs remains open.

Target Audience

Researchers and engineers working on multimodal foundation models, hallucination mitigation, and efficient LLM inference. It is most useful to readers already comfortable with transformer attention mechanics, decoding-time intervention methods, and benchmark conventions such as POPE, CHAIR, AMBER, MME, and MM-Vet. Practitioners deploying LVLMs under latency and memory constraints will find the efficiency results and the drop-in, training-free design directly actionable, while newcomers to the area may need the related-work section for background on contrastive and attention-guided decoding baselines.

Authors’ abstract

Large vision-language models have achieved remarkable success in vision-language tasks. However, they remain prone to Visual Hallucinations (VHs), undermining their reliability in real-world applications. Existing solutions typically require curated datasets, additional training, or multi-round decoding, resulting in considerable computational overhead. In this paper, we propose \textbf{RVSD} (\underline{R}etrieval \underline{V}ision \underline{S}parse \underline{D}ecoding), a training-free and plug-and-play decoding framework that, for the first time, unifies token sparsification and \textbf{Semantic-Space Visual Retrieval} (SSVR) within a single decoding pass. Within RVSD, we introduce a \textbf{semantics-directed token selection} strategy that selectively sparsifies redundant tokens while preserving critical visual information. We further propose the SSVR mechanism, which reformulates visual compensation as an on-demand cross-modal retrieval process within a shared semantic space. Extensive experiments demonstrate that RVSD achieves state-of-the-art performance in mitigating VHs while maintaining robust suppression capabilities under long-context generation settings. Our code is available here.\footnote{https://github.com/canjie-liu/RVSD}