NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors

AI summary

Overview

Research area: Computer Vision / Vision-Language Models — specifically object hallucination in Large Vision-Language Models (LVLMs), with a focus on inference-time decoding strategies.

Technical level: Intermediate. The paper is readable with a basic understanding of how LVLMs generate text token-by-token and what a probability distribution (logits) over a vocabulary means. The core method is conceptually simple; the appendix contains a theoretical derivation.

Scope: The paper diagnoses whether the vision encoder or the language decoder causes object hallucinations, then proposes a training-free decoding method, NoLan, that suppresses language priors at generation time.

What This Paper Is About

Large Vision-Language Models sometimes describe objects that are not in the image — a failure called object hallucination. The authors ask which part of the pipeline is responsible: the vision encoder that perceives the image, or the language decoder that writes the answer. They conclude the language decoder's learned textual priors are the main culprit, and they build a decoding rule that dampens those priors during generation.

Key Contributions

1. A diagnostic analysis separating the roles of the vision encoder and language decoder. The authors run two sets of experiments: one testing whether a CLIP vision encoder alone can detect object presence in cases where LLaVA-1.5 hallucinates, and one comparing output token distributions from multimodal versus text-only inputs.

2. NoLan (No-Language-Hallucination Decoding), a training-free, plug-and-play decoding framework. It contrasts the logits from multimodal input with the logits from text-only input and adds the difference back into the output distribution, requiring no additional training, datasets, or external pretrained models.

3. Two variants with different modulation strategies. NoLan-Base uses a fixed modulation rate (α = 1 by default); NoLan-Plus uses a self-adjusting rate derived from the symmetric KL divergence between the multimodal and text-only distributions, scaled by β = 0.8.

4. Extensive evaluation across multiple LVLMs and hallucination benchmarks, including LLaVA-1.5, InstructBLIP, and Qwen-VL on POPE (MSCOCO, A-OKVQA, GQA splits), the MME hallucination subset, and LLaVA-Bench.

Main Findings

• Finding 1 — The vision encoder still works during hallucinations. On the MSCOCO portion of POPE-random, restricted to samples where LLaVA-1.5 hallucinates, its CLIP vision encoder predicted object presence with accuracy 83.01, precision 83.71, recall 98.33, and F1 90.43 (the text describes this as a high accuracy of 83%). This suggests the visual signal is available even when the output is wrong.

• Finding 2 — Hallucinations coincide with language-prior dominance. Splitting MSCOCO/POPE-random by whether LLaVA-1.5-7B hallucinated: the no-hallucination subset had D_KL(p_m ‖ p_u) = 1.20, D_KL(p_u ‖ p_m) = 0.58, and D_JS(p_m, p_u) = 0.28, while the hallucination subset had 0.46, 0.28, and 0.11. Smaller divergence between multimodal and text-only distributions goes together with hallucination, meaning the text-only prior is steering the output.

• POPE results. NoLan improved accuracy by up to 8.38 and F1 by up to 8.77 across the tested LVLMs relative to regular decoding (the abstract separately reports accuracy gains of up to 6.45 for LLaVA-1.5 7B and 7.21 for Qwen-VL 7B, and an F1 improvement of up to 8.78 in the introduction). NoLan-Base beat VCD by up to 4.56 accuracy and 2.9 F1, outperforming VCD in 77.8% of evaluated cases; NoLan-Plus beat VCD by up to 5.14 accuracy and 3.17 F1, outperforming it in 88.9% of cases. Selected examples: on GQA-random, LLaVA-1.5 went from 83.73 (Regular) and 86.65 (VCD) to 88.35 (NoLan-Base) and 88.53 (NoLan-Plus) accuracy.

• Comparison with M3ID. Using M3ID's template, NoLan suppressed M3ID in accuracy by up to 18.2 and 13.9, with average increases of 16.4 and 10.1 on the 7B and 13B models respectively. For instance, on MSCOCO/POPE "All," LLaVA-1.5-7B moved from 64.9 (Regular) and 70.3 (M3ID) to 85.6 (NoLan-Base) and 86.7 (NoLan-Plus).

• Comparison with VDD. Under VDD's decoding format, NoLan-Plus still outperformed VDD on both the 7B and 13B models (for example, on MSCOCO/POPE "All," LLaVA-1.5-13B: Regular 81.92, VDD 85.77, NoLan-Plus 87.00), while NoLan-Base was slightly below VDD in some settings.

• MME hallucination subset. NoLan improved attribute-level hallucination scores across models. Total scores: LLaVA-1.5 Regular 565.33, VCD 604.66, VDD 643.29, NoLan-Base 628.33, NoLan-Plus 660.00; Qwen-VL Regular 587.33, VCD 596.67, VDD 643.29, NoLan-Base 618.33, NoLan-Plus 648.33; InstructBLIP Regular 380.33, VCD 447.67, NoLan-Base 423.33, NoLan-Plus 460.00. On the Existence metric, both NoLan variants had a clear positive effect.

• Qualitative case studies on LLaVA-Bench showed NoLan-Plus removing hallucinated objects such as "suitcase" and "truck" that were spuriously associated with the ground-truth object "taxi," while keeping the generated text informative.

Methodology in Plain English

The authors start by isolating the two halves of an LVLM. To test the vision side, they take the CLIP image encoder from LLaVA and, for images where LLaVA hallucinated, check whether the image representation is close to a text representation of "A photo of a [object]" using cosine similarity. To test the language side, they feed the model the same question with and without the image, and compare the two next-token probability distributions using KL and JS divergence.

Because the results point at language priors, the fix works on the output distribution. At each generation step, the model produces two sets of logits: one from the image plus question (l_m), and one from the question alone (l_u). The method takes their difference, scales it by a modulation rate α, and adds it back: the final distribution is softmax(l_m + α(l_m − l_u)). Intuitively, tokens that the model would have produced anyway from text alone get penalized, while tokens that the image specifically supports get boosted.

NoLan-Base simply fixes α at 1. NoLan-Plus computes the symmetric KL divergence between the two distributions, inverts it, passes it through a tanh function, shifts it by 1, and scales by β = 0.8. This makes the correction stronger when the two distributions are similar (the situation the paper associates with hallucination) and weaker when they already differ. Generation then proceeds normally with standard sampling such as top-p or beam search. No retraining, fine-tuning, or extra models are involved.

Why This Matters

Impact on research. The paper reframes object hallucination as primarily a language-prior problem rather than a visual-perception failure, and it argues for a finer-grained assumption than prior training-free methods: VCD and VDD treat the language prior as uniform across tokens, and M3ID conditions it only on sequence length, whereas NoLan treats each token as having its own distinct language prior measured by a KL-based quantity. It also raises the bar for training-free baselines, since NoLan outperforms VCD in a large majority of reported comparisons without external tools.

Real-world applications.

• Robotics and embodied agents, where a hallucinated object could trigger a wrong physical action.
• Autonomous systems, where misidentified objects in a scene carry safety risk.
• Healthcare, where misdescribed medical images could mislead clinical interpretation.
• Content generation, image and video annotation, and interactive platforms that rely on accurate visual description.

Industry relevance. The method is deployed at inference time and is described as compatible with any pre-trained autoregressive LVLM, so it can be dropped into existing products without the cost of collecting hallucination-specific datasets, running RLHF, or maintaining auxiliary reviser models. That makes it attractive for teams that need to reduce hallucination without retraining large models.

Future Directions

• Confirming the mechanism more broadly. The diagnostic experiments focus on LLaVA-1.5 (with a CLIP encoder and LLaMA/Vicuna decoder); whether the same language-prior dominance holds across other encoder-decoder pairings is an open question the paper does not settle.
• Tuning the modulation. NoLan-Plus depends on the scaling factor β = 0.8, selected from ablation experiments; how robust that choice is across models, tasks, and longer generations is not fully resolved.
• Scaling to broader hallucination types. The evaluation centers on object- and attribute-level hallucination benchmarks (POPE, the MME hallucination subset, LLaVA-Bench, with MM-Vet, MMHal-Bench, and HallusionBench detailed in the appendix); extending the analysis to more complex reasoning-level hallucinations remains future work.
• Consistency of reported gains. The paper reports different headline improvement numbers in the abstract, introduction, and results sections (up to 6.45 and 7.21 accuracy in the abstract versus up to 8.38 accuracy and 8.77 F1 in the results), which points to a need for clearer standardized reporting across settings.

Target Audience

Researchers and engineers working on multimodal foundation models, hallucination mitigation, or decoding-time interventions will get the most from this paper. It is also useful for practitioners who need a training-free method they can attach to an existing LVLM, and for students who want a worked example of diagnosing a model failure by decomposing a system into its components before designing a fix.

Authors’ abstract

Object hallucination is a critical issue in Large Vision-Language Models (LVLMs), where outputs include objects that do not appear in the input image. A natural question arises from this phenomenon: Which component of the LVLM pipeline primarily contributes to object hallucinations? The vision encoder to perceive visual information, or the language decoder to generate text responses? In this work, we strive to answer this question through designing a systematic experiment to analyze the roles of the vision encoder and the language decoder in hallucination generation. Our observations reveal that object hallucinations are predominantly associated with the strong priors from the language decoder. Based on this finding, we propose a simple and training-free framework, No-Language-Hallucination Decoding, NoLan, which refines the output distribution by dynamically suppressing language priors, modulated based on the output distribution difference between multimodal and text-only inputs. Experimental results demonstrate that NoLan effectively reduces object hallucinations across various LVLMs on different tasks. For instance, NoLan achieves substantial improvements on POPE, enhancing the accuracy of LLaVA-1.5 7B and Qwen-VL 7B by up to 6.45 and 7.21, respectively. The code is publicly available at: https://github.com/lingfengren/NoLan.