Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models

AI summary

Overview

• Research area: Computer Vision / multimodal NLP — specifically training-free decoding strategies for reducing hallucination in Large Vision-Language Models (LVLMs).
• Technical level: Intermediate. The paper assumes familiarity with auto-regressive decoding, logits, softmax distributions, Shannon entropy, and contrastive decoding (CD/VCD).
• Scope: The paper proposes two decoding-time components — a self-augmentation prompting strategy that picks a query-relevant visual perturbation, and an entropy-based token truncation rule — and evaluates them across five LVLMs and seven benchmarks.

What This Paper Is About

LVLMs hallucinate: they produce text that sounds plausible but is not faithful to the image or the user's question. Contrastive decoding tries to fix this by comparing a model's output on the real image against its output on a deliberately degraded version of the same image, but most existing methods choose that degradation without regard to what the text query actually asks. This paper's goal is to let the model itself pick the visual augmentation best matched to the query, and to make the candidate-token cutoff aware of how confident the model is at each generation step.

Key Contributions

1. Self-Aug (Self-Augmentation Selection, SAS): A prompting strategy that uses the LVLM's own parametric knowledge — rather than a fixed heuristic or brute-force search — to select a visual augmentation that is semantically aligned with the text query, producing a more informative expert/amateur discrepancy.
2. Sparsity Adaptive Truncation (SAT): A replacement for the Adaptive Plausibility Constraint (APC) that sets the token-filtering threshold from the entropy of the full expert logit distribution instead of only its maximum value, making the cutoff confidence-aware.
3. Training-free integration: The method requires no architectural modification and no additional training, and is applied to any LVLM.
4. Large-scale empirical validation: Experiments across five LVLMs (LLaVA-1.5-7B/13B, Qwen-VL, InstructBLIP, Qwen3-VL-8B) and seven benchmarks, plus ablations on augmentation choice, prompt components, thresholding functions, and computational overhead.

Main Findings

• Overall gains: Self-Aug reports average performance gains over multinomial sampling of up to 18.78% relative, achieved on the InstructBLIP discriminative setting. Other per-model average deltas in Table 1 and Table 2 include LLaVA-1.5-7B at +14.32% (discriminative) and +6.97% (generative), LLaVA-1.5-13B at +11.59% and +6.04%, InstructBLIP at +17.08% (generative), Qwen-VL at +17.09% (generative), and Qwen3-VL-8B at +2.25% and +2.62%.
• Results are not uniformly best: On Qwen-VL's discriminative average, Self-Aug scores +6.69% versus VACoDe's +7.49%. On Qwen-VL's MMHal-Bench average score, Self-Aug records 2.15 ±0.06 while VACoDe records 2.21 ±0.13.
• Strongest absolute discriminative result: Qwen3-VL-8B reaches MME-P 1726.77 ±3.63 and MMVP 60.50 ±0.64 with Self-Aug, the highest MMVP figure reported in the paper.
• Hallucination rate reduction: On MMHal-Bench, Self-Aug lowers the hallucination rate to 0.59 ±0.03 for LLaVA-1.5-7B, 0.60 ±0.03 for LLaVA-1.5-13B, 0.64 ±0.05 for InstructBLIP, and 0.29 ±0.01 for Qwen3-VL-8B.
• Query-dependent augmentation matters: Static augmentations on MME-P with LLaVA-1.5-7B range from 1302.50 ±29.34 (random mask) to 1354.42 ±15.13 (vertical flip), while adaptive selection reaches 1431.30 ±13.87 (Self-Aug) and 1435.07 ±22.30 (Oracle, i.e., gpt-4o-mini choices).
• SAT beats APC consistently: Applying SAT instead of APC (β = 0.1) improves MME-P for VCD (1323.67 ±20.84 → 1395.17 ±17.09), VACoDe (1372.50 ±13.78 → 1414.21 ±26.85), and Self-Aug (1345.46 ±4.65 → 1431.30 ±13.87), an average gain of 4.94%.
• Inverse-entropy heuristic confirmed: The proposed decayed entropy function H_decay outperforms both APC and the directly-proportional normalized scaled entropy H_ns, supporting the claim that higher entropy should yield a more lenient threshold.
• Operational knowledge dominates the prompt: In the SAS prompt ablation, baseline MME-P is 1278.42 ±30.30; a prompt with none of the components gives 1312.39 ±11.71; operational knowledge alone gives 1419.08 ±11.39; adding reasoning gives 1428.02 ±7.92; adding ICL gives 1428.63 ±31.85; all three give 1431.30 ±13.87.
• Bigger models choose better augmentations: Agreement with the gpt-4o-mini "Oracle" choice is 64.15% for LLaVA-1.5-7B versus 66.19% for 13B, and gpt-4o-mini rates the reasoning traces 8.28/10 (7B) versus 9.04/10 (13B).
• Augmentation choice is benchmark-dependent: Random mask accounts for 87.6% of selections on POPE but the distribution is most uniform on MMVP; horizontal flip is rarely selected across benchmarks.
• Latency trade-off: On LLaVA-Bench with LLaVA-1.5 on an NVIDIA A100, the full Self-Aug configuration reaches 15.08 tok/s (66.32 ms/token) at 7B and 11.33 tok/s (88.26 ms/token) at 13B, versus VCD at 18.50 tok/s (54.06 ms/token) at 7B. A lightweight variant without reasoning and ICL runs at 17.39 tok/s (57.50 ms/token) at 7B.

Methodology in Plain English

The method has two stages, both applied at inference time only.

Stage one — choosing how to degrade the image. Before generation begins, the LVLM is given a structured prompt describing six possible image augmentations (random crop, random mask, noise addition, color inversion, horizontal flip, vertical flip) along with what each does. The prompt asks the model to first reason about the query, then name the single augmentation that would most disrupt the visual evidence needed to answer it. A parser splits the model's output into a reasoning trace and the final choice, and that choice is applied to the image to create the "amateur" input. Because this prompt is text-only and uses greedy decoding, it avoids a full visual forward pass per candidate augmentation.

Stage two — decoding with contrast. At each generation step, the method computes the expert logit (on the original image) and the amateur logit (on the augmented image) and combines them contrastively with amplification parameter α. To stop the subtraction from promoting implausible tokens, it first truncates the vocabulary. Instead of a fixed threshold derived from the maximum logit (APC), it computes Shannon entropy over the softmax of the expert distribution and passes it through a sigmoid of the negated scaled entropy, with scaling parameter γ. High entropy produces a smaller threshold (more candidates retained); low entropy produces a larger threshold (fewer candidates). The threshold is bounded strictly below 1 so the candidate set never collapses to a single token. Tokens outside the set are assigned negative infinity.

Evaluation setup. Discriminative benchmarks (POPE on MSCOCO, POPE on A-OKVQA, MME-Perception, MMVP) test factuality through binary or multiple-choice questions; generative benchmarks (MMHal-Bench, LLaVA-Bench In-the-Wild, MM-Vet) require open-ended answers judged by proprietary models. Hyperparameters were α = 1, β = 0.1 for APC, and γ = −0.5 for SAT. All proprietary-model API calls, including generative judging, used OpenAI gpt-4o-mini at temperature 0. Main experiments were run five times and ablations three times with different random seeds, with results reported as mean and standard deviation.

Why This Matters

The paper's central claim is that which image perturbation you contrast against should depend on what is being asked, and that the token cutoff should depend on how confident the model is. Both operate purely at decoding time, so they can be bolted onto existing systems without retraining — attractive for practitioners who cannot fine-tune large multimodal models.

Potential real-world applications (implied by the paper's framing rather than demonstrated):

• Visual question answering and image captioning assistants where factual errors about image content are costly.
• Document, chart, or handwriting interpretation tasks where the query type varies sharply and a single fixed augmentation cannot serve all cases.
• Accessibility tools that narrate images, where unfaithful descriptions mislead users.
• Any deployed multimodal assistant that needs real-time hallucination control without a retraining pipeline.

Industry relevance: The computational analysis matters commercially — the paper argues Self-Aug's single text-only generation pass is architecturally cheaper than VACoDe's brute-force forward pass per augmentation, giving a tunable accuracy/latency dial. The reported gains shrink on the strongest model tested (Qwen3-VL-8B, +2.25% and +2.62%), which is a relevant signal for teams already deploying capable models.

Future Directions

1. Robustness on weaker models: The authors note that SAS prompting depends on the base model's reasoning and instruction-following ability, and that less capable models may produce malformed outputs or poor augmentation choices. They suggest Chain-of-Thought prompting or an external selection module as possible remedies.
2. Addressing contrastive decoding failure cases: The paper formalizes when CD assumptions break down (when the amateur logit is not actually worse on false positives, or not actually better on true positives) and demonstrates this on Qwen3-VL-32B, but explicitly defers solving it to future work.
3. Richer prompt and augmentation design: The authors state they limited scope to two prompting features and six augmentations, noting that more optimized prompting techniques and combinations of augmentations remain unexplored.
4. Extending SAT beyond this method: Because SAT improved VCD, VACoDe, and Self-Aug alike, the authors suggest it is broadly applicable to other contrastive decoding methods.

Target Audience

Researchers and engineers working on multimodal model inference, hallucination mitigation, and decoding algorithms. It is most useful to readers who already understand logits, softmax, and entropy-based uncertainty, and who want a training-free decoding intervention they can layer onto an existing LVLM pipeline. Readers seeking a new architecture or a fine-tuning recipe will not find one here.

Authors’ abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal capabilities, but they inherit the tendency to hallucinate from their underlying language models. While visual contrastive decoding has been proposed to mitigate this issue, existing methods often apply generic visual augmentations that disregard the specific context provided by the text query, limiting their effectiveness. This study introduces a novel training-free decoding strategy that addresses these limitations, featuring two key contributions. First, a self-augmentation prompting strategy that leverages the intrinsic knowledge of the model to dynamically align semantics between the query and the visual augmentation. Second, an adaptive thresholding algorithm that adaptively adjusts next token candidate size based on the output sparsity, utilizing full information from the logit distribution. Extensive experiments across four LVLMs and seven benchmarks demonstrate that the proposed decoding significantly enhances factual consistency compared to state-of-the-art decoding methods. This work highlights the importance of integrating query-dependent augmentation and entropy-aware decoding for improving effective generation of LVLMs.