Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidance

AI summary

Overview

Research area: Computer Vision / multimodal large language models — specifically hallucination mitigation at inference time in Large Vision-Language Models (LVLMs).

Technical level: Advanced. The paper builds on information-theoretic divergence measures (Jensen–Shannon Divergence, KL divergence) and probabilistic decompositions of language priors, though the resulting method is a simple drop-in decoding change.

Scope in one sentence: The paper introduces Residual Decoding (ResDec), a training-free, plug-and-play decoding strategy that uses a model's own historical token logits to suppress language-prior hallucinations, evaluated across eleven benchmarks and three LVLMs.

What This Paper Is About

LVLMs frequently generate fluent, grammatically correct text that has no grounding in the image they were shown — a failure the authors call Language-Prior Hallucination. The paper argues that the root cause is visible at decoding time: hallucinated tokens surface in the logit distribution at certain steps and their probabilities creep up to or past those of genuine tokens, so the model emits the wrong token. The goal is to correct this during generation without retraining the model, adding a contrastive model, or paying a large inference cost.

Key Contributions

1. A decoding-time diagnosis of hallucination. The authors show that signals for the correct answer already sit at relatively high logit values in the distributions of preceding tokens (e.g., while generating "The", "answer", "is"), and that hallucination occurs when a wrong token's logits spike at a later step and overtake the genuine one.

2. Residual Decoding (ResDec). A training-free decoding strategy that computes Jensen–Shannon Divergence (JSD) between the logit distributions of adjacent time steps in a sliding historical window, identifies a U-shaped JSD curve, and aggregates logits from the post-convergence region into a "residual guidance" stream that is blended into the current decoding logits.

3. A three-phase account of logit evolution. The paper decomposes the U-shape into the Pre-Semantic Clarity Phase (PSAP), the Semantic Anchoring Phase (SAP) at the JSD valley, and the Expressive Divergence Phase (EDP), and selects history from SAP and EDP.

4. Broad empirical validation. ResDec is tested on eleven benchmarks across LLaVA-1.5 (7B), Qwen2.5-VL-Instruct (7B), and InstructBLIP (Vicuna-7B), reporting improvements over regular decoding and over eight compared methods (VCD, ICD, DoLa, OPERA, AGLA, ONLY, MemVR, VISTA).

Main Findings

• Hallucinated tokens win the logit race. The observed failure mode is not absence of evidence but late-arriving spurious logits that push a hallucinated token above the correct one at the moment of explicit decoding.

• JSD follows a U-shape over generation time. Computed by averaging over 200 randomly sampled examples from POPE-MSCOCO using Qwen2.5-VL-Instruct (7B), JSD falls into a valley (highest distributional stability, minimal divergence between consecutive time steps) and then rises again.

• Three phases, not one. From onset to the valley is PSAP (fluctuating downward, "disorder" to "convergence"); at the valley is SAP (near-zero JSD); from the valley to step t is EDP (rising JSD as the model pursues templated and diverse expression, and where susceptibility to language priors grows).

• Implicit answers strengthen as context fills in. In Figure 2 (Qwen2.5-VL 7B on MME and POPE, both binary Yes/No), accuracy of candidate answers rises as the question is described more completely, peaking near the token to be decoded — but this implicit representation is unstable because of grammar preferences and task-specific template preferences.

• POPE results. ResDec improves over "Regular" decoding by an average of 7.84% in accuracy and 8.01% in F1 across the three LVLMs and settings, and outperforms all compared SOTA methods.

• HallusionBench and CHAIR results. ResDec leads across all three models on HallusionBench; on CHAIR, relative to regular decoding, it yields average improvements of 16.57% on CHAIR_S and 26.44% on CHAIR_I. CHAIR evaluation capped generation at 128 new tokens.

• Comprehensive benchmarks. Across the three models, ResDec outperforms "Regular" decoding by 2.87% on MME, 2.99% on ScienceQA, and 11.65% on MMVP. Example absolute numbers: on Qwen2.5-VL, MME rises from 2309.42 (Regular) to 2348.40, MMVP from 58.00 to 63.33, and MM-Vet from 67.10 to 68.70.

• Candidate pool size matters. Sweeping pool sizes from 2 to 4096 tokens (on LLaVA-1.5, POPE and MME) shows the best range is 64 to 512; too small a pool fails to capture JSD variation, too large a pool distorts the divergence metric and mislocates SAP.

• Hyperparameters. Defaults are α = 0.5 and β = 0.1. α values of 0.75 and 1.0 degrade performance (the historical signal is auxiliary, not a replacement). β = 0 is suboptimal while β > 0 improves results, validating the truncation constraint.

• Robust across sampling strategies. On POPE and MME with LLaVA-1.5, ResDec improves results under Nucleus Sampling (p = 0.7), Top-K sampling (k = 50), Temperature Sampling (t = 0.5), and Greedy decoding. For example, Greedy goes from 79.83 accuracy / 79.29 F1 on POPE to 87.23 / 86.93.

• Low overhead. ResDec reuses logits already produced during inference. Reported latency is 29.11 ms/token versus 28.54 for Greedy (described as only 0.02× slower than Greedy decoding), throughput 34.35 token/s versus 35.04, and memory 14296 MB versus 14257 MB. For comparison, OPERA is listed at 104.46 ms/token and VCD at 62.79 ms/token.

• Qualitative case study. Figure 5 shows regular decoding adding details absent from the images (additional people, specific fruit types) that ResDec removes while keeping captions coherent.

Methodology in Plain English

The authors start from a probabilistic view of language priors: the text-only, vision-agnostic distribution P(y | x) can be "divided out" of the visually conditioned prediction P(y | v, x), leaving a ratio analogous to pointwise mutual information that measures how much the image supports an output beyond the text prior. They also approximate the language prior by Monte Carlo averaging over sampled images, though ResDec itself does not require this sampling.

The practical method is simpler than that math suggests. During normal autoregressive generation, the model already produces a logit distribution at every step. ResDec keeps a sliding history window of these distributions and measures the Jensen–Shannon Divergence between each adjacent pair. The resulting U-shaped curve tells the method where semantics have stabilized: it takes the bottom of the valley plus the region after it (SAP and EDP), avoids the noisy early PSAP region, and forms a candidate token pool from the top-k highest-probability tokens at the current step.

Each historical step in that window is weighted by a confidence score derived from the negative average log-probability of the pool tokens — essentially a local entropy measure, so steps where the model is more certain contribute more. These weighted historical logits form a residual signal, which is combined with the current step's logits via a weighted sum controlled by α, then passed through softmax. A second hyperparameter, β, applies an adaptive plausibility constraint: tokens whose probability falls below β times the maximum probability are set to negative infinity and filtered out. The whole procedure needs a single forward pass, no extra model, and no external encoder.

Why This Matters

Impact on research. The paper reframes hallucination as a decodable, observable event in the logit stream rather than only a property of training data or alignment. If the U-shaped JSD signature and three-phase decomposition hold more broadly, they give the field a cheap diagnostic and a new class of decoding-time interventions that sidestep the cost and fragility of contrastive decoding and architectural surgery.

Real-world applications.

• Assistive technology: image description for blind and low-vision users, where invented objects (the "additional people" and "specific fruit types" in the case study) can be actively misleading.
• Medical imaging and clinical reporting, where ungrounded findings carry direct safety consequences.
• E-commerce and catalog automation, where hallucinated product attributes in generated descriptions cause downstream errors.
• Robotics and autonomous systems, where scene descriptions that ignore the actual visual input can corrupt downstream planning.
• Content moderation and archival captioning at scale, where low-latency decoding matters.

Industry relevance. The method needs no retraining, no annotation, and no extra model, and it reuses logits that are already computed. That means it can be applied to an already-deployed LVLM as a decoding configuration change — an attractive proposition for teams that cannot afford fine-tuning or doubled inference costs.

Future Directions

• Automating window selection. The historical window W is set empirically and the candidate pool size must be tuned (64 to 512 works best here); reliably locating SAP at runtime, rather than within a fixed range, is an open problem the paper leaves to an appendix discussion.
• Robustness of α. Performance degrades sharply at α = 0.75 and α = 1.0, so the balance between current and historical logits appears regime-sensitive; understanding why, and whether α should adapt during generation, is unresolved.
• Beyond the tested configurations. The evaluation covers three 7B-class models and text-image inputs; whether the U-shape and the method transfer to larger models, other architectures, video, or audio-visual inputs is not reported.
• Combination with training-based approaches. The paper positions ResDec against data debiasing, modality alignment, and preference alignment rather than combining with them; whether decoding-time residual guidance complements those methods is untested.

Target Audience

Researchers and practitioners working on multimodal model reliability, hallucination evaluation, and inference-time decoding strategies. It is most useful to engineers who need a low-overhead fix for an already-deployed LVLM, and to researchers interested in the internal dynamics of autoregressive multimodal generation. Readers without a background in information theory or language model decoding will need to work through the JSD and divergence material, though the core idea — reuse earlier, more stable logits to correct the current one — is graspable without it.

Authors’ abstract

Large Vision-Language Models (LVLMs) can reason from image-text inputs and perform well in various multimodal tasks. Despite this success, they are affected by language priors and often produce hallucinations. Hallucinations denote generated content that is grammatically and syntactically coherent, yet bears no match or direct relevance to visual input. To address this problem, we propose Residual Decoding (ResDec). It is a novel training-free method that uses historical information to aid decoding. The method relies on the internal implicit reasoning mechanism and token logits evolution mechanism of LVLMs to correct biases. Extensive experiments demonstrate that ResDec effectively suppresses hallucinations induced by language priors, significantly improves visual grounding, and reduces object hallucinations. In addition to mitigating hallucinations, ResDec also performs exceptionally well on comprehensive LVLM benchmarks, highlighting its broad applicability.