Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding

AI summary

Overview

Research area: Medical computer vision and multimodal AI — specifically hallucination mitigation for medical large vision-language models (Med-LVLMs) used in visual question answering (VQA) and radiology-style report generation.

Technical level: Intermediate. The paper assumes familiarity with LVLM architecture (vision encoders, vision-text alignment modules, autoregressive decoders, key-value caching), attention mechanics, and contrastive decoding, though the core idea can be grasped without implementing the optimization framework.

Scope: The paper proposes Med-VCD, a training-free, plug-and-play decoding method that combines visual-aware token selection, sparse contrastive decoding, and attention calibration to reduce hallucinations in medical LVLMs while avoiding the cost of secondary decoding.

What This Paper Is About

Medical vision-language models frequently produce hallucinated output — text that reads plausibly but is factually wrong, such as inventing a "pleural effusion" finding that is not in the image. Existing fixes either require expensive retraining on scarce high-quality labeled medical data, or rely on visual contrastive decoding (VCD) methods that run multiple decoding rounds and external visual localization tools, which slows inference badly. Med-VCD aims to reduce medical hallucinations without retraining and without the time overhead of secondary decoding, by selecting only the visually important tokens during generation.

Key Contributions

1. Reframing hallucination mitigation as token sparsification. The authors investigate hallucination reduction from the standpoint of token sparsification during decoding and cast the joint goal of efficiency and fidelity as a single constrained optimization problem.

2. Visual-aware token selection (VATS). A new selection strategy ranks tokens per attention head by an aggregated score combining attention relevance and a visual saliency score, retaining the top-S tokens and pruning the rest.

3. Sparse-based VCD without multi-round decoding. A contrastive decoding scheme that derives the contrastive logit distribution by passing only vision-agnostic token embeddings through the language decoding head, bypassing the Med-LVLM decoder entirely and avoiding full autoregressive recomputation.

4. Sinking Attention Calibration (SAC). A penalty mechanism for tokens that attract disproportionately high attention despite minimal semantic relevance, incorporated into the attention computation.

5. Broad evaluation. Tests on three hallucination benchmarks and two Med-LVLM architectures, reporting gains in both performance and decoding speed over existing methods.

Main Findings

• Aggregate accuracy gains: On eight medical datasets spanning ophthalmology, radiology, and pathology — across visual question answering, report generation, and dedicated hallucination benchmarks — Med-VCD raises factual accuracy by an average of 13% and improves hallucination accuracy by 6% relative to baseline medical LVLMs.

• Best-in-class on medical VQA with LLaVA-Med-1.5 (Table 1): Med-VCD reaches 90.56 accuracy / 80.30 F1 / 88.03 AUROC on IU-Xray; 82.95 / 89.38 / 85.42 on MIMIC-CXR; 88.73 / 92.58 / 81.36 on Harvard-FairVLMed; 73.52 / 75.60 / 72.96 on Quilt-1M; and 65.31 / 74.19 / 62.38 on PMC-OA (Pathology). The authors state Med-VCD performs better than all baselines in almost every metric and dataset.

• Report generation gains (Table 2): On IU-Xray, Med-VCD scores 32.59 BLEU, 25.63 ROUGE-L, 31.20 METEOR; on MIMIC-CXR, 24.12 / 15.89 / 21.95; on Harvard-FairVLMed, 23.15 / 16.79 / 18.13.

• Headline comparisons reported in Section 4.2: Med-VCD outperforms the original Med-LVLM by 25.7% and 22.9% on medical VQA and report generation respectively; outperforms decoding-based methods by 22.8% and 22.2% on the same two tasks; and beats RAG-based baselines by 5% and 17.8%.

• Versus other open-source Med-LVLMs (Table 3): Med-VCD scores 57.74 (Radiology), 53.90 (Ophthalmology), 55.39 (Pathology), compared with Med-Flamingo (27.42 / 22.50 / 29.11), MedVInT (33.17 / 29.40 / 25.33), RadFM (35.82 / 27.07 / 24.82), and miniGPT-Med (36.66 / 25.28 / 23.16).

• Closed-ended visual misinterpretation hallucination (Table 4): On LLaVA-Med-1.5, Med-VCD achieves 0.671 Acc-A, 0.403 Acc-M, 0.554 Acc-S, 0.528 Acc-R, 0.549 Acc on MM-VisHal, and 0.883 / 0.539 / 0.755 / 0.886 / 0.766 on CXR-VisHal — the best across most settings compared with VCD, DoLa, OPERA, AVISC, M3ID, DAMRO, and PAI.

• Open-ended visual misinterpretation (Table 5): With LLaVA-Med, Med-VCD achieves CheXbert 24.50, RadGraph 8.97, RaTEScore 41.33, Recall 32.74, and CHAIR 15.02 — the lowest CHAIR (better) among the compared methods on that backbone.

• Knowledge deficiency hallucination: Unlike visual-misinterpretation tasks, only Med-VCD and OPERA yield consistent improvements on the closed-ended evaluation (Figures 5a and 5b). On the open-ended benchmark (Table 6), Med-VCD with LLaVA-Med-1.5 scores BertScore 90.35, BLEU 14.96, METEOR 42.98, ROUGE-1 44.41, ROUGE-2 21.83, ROUGE-L 32.62, and a hallucination score of 1.39 ± 0.91. The authors note improvements here are modest.

• Context misalignment hallucination: On the LLaVA-Med benchmark, only Med-VCD demonstrates meaningful performance improvement over existing mitigation methods (Figure 6a and 6b); the discussion of limits is cut off in the available text.

• Method generality: Med-VCD reduces hallucinations across two distinct Med-LVLM architectures (LLaVA-Med and LLaVA-Med-1.5), which the authors present as evidence of plug-and-play versatility.

Methodology in Plain English

The observation. When the authors sorted tokens by attention score in LLaVA-Med-1.5, they found a long-tailed distribution: only a small fraction of tokens receives substantial attention, and most decoder layers follow sparse self-attention patterns. A separate analysis of attention density showed image tokens mostly sit in low-attention regions while text tokens concentrate in high-attention regions — and some low-attention tokens are still critical for interpreting the image. Naively pruning them makes visual hallucination worse.

The optimization framing. Token removal is formalized with a binary mask over the key-value cache: a mask value of 0 prunes a token, 1 keeps it. The objective balances two things — how closely the sparse attention score approximates the full attention score, and a visual saliency term weighted by a tradeoff parameter — subject to keeping exactly S tokens.

Selecting what to keep (VATS). For each attention head, tokens are ranked by an aggregated score that adds the squared attention relevance to the scaled visual saliency score. The top-S are kept; the rest are pruned. The visual saliency scores are computed from attention weights already produced in earlier layers between generated tokens and image tokens, so no extra computation is introduced. Pruned tokens are not simply discarded — they are merged within clusters using a k-nearest neighbor density peak clustering algorithm, compressing redundancy while keeping salient features.

Getting a contrast without a second decoding pass. Standard contrastive decoding compares two full forward passes, which doubles cost. Med-VCD instead compares the vision-aware sparsification distribution against a vision-agnostic, stochastically masked distribution, and obtains the contrastive distribution by feeding only the vision-agnostic token embeddings into the language decoding head — skipping the decoder. The final token probability is a weighted fusion of the primary and contrastive distributions, with adaptive plausibility constraints added to keep selections linguistically coherent and contextually grounded.

Suppressing attention sinks. Certain tokens attract attention scores disproportionate to their semantic relevance and can overshadow visual information. Med-VCD adds a learnable penalty weight matrix whose entries are softmax-normalized cumulative attention scores, then applies a transformation to the attention computation that scales down over-activated tokens while preserving attention to visually salient patterns.

Setup. Backbones: LLaVA-Med and LLaVA-Med-1.5 7B. Implemented with the Transformer library and integrated with beam search. Maximum generation length 1024 tokens, beam size 2, sparsity rate 0.8 times the sequence length, image masking sparsity rate 0.5, and hyperparameters set to 0.1, 0.3, and 0.1. Experiments run on A100 GPUs. VQA benchmarks were built by using GPT-4 to generate yes/no question-answer pairs from medical reports; pathology images were excluded from report generation because their explanations are short and inadequate.

Why This Matters

Impact on research. The paper demonstrates that hallucination mitigation and inference efficiency need not be traded off against each other in the medical domain, and that a training-free decoding intervention can rival or beat retrieval-augmented and fine-tuning-based alternatives. It also shows that token sparsity — normally a speed optimization — can be repurposed as a source of contrastive signal rather than a source of error.

Real-world applications:

• Medical visual question answering, where clinicians or triage systems query an image and need a reliable yes/no or short-form answer.
• Radiology report generation from chest X-rays, evaluated here with BLEU, ROUGE-L, METEOR, CheXbert, RadGraph, RaTEScore, Recall, and CHAIR.
• Ophthalmology and pathology image interpretation, covered by the Harvard-FairVLMed and PMC-OA (Pathology) datasets.
• Context-aware clinical interpretation that links imaging to patient history, tested by joining MIMIC-CXR with the de-identified MIMIC-IV EHR dataset by subject ID.

Industry relevance. Because the method requires no retraining and no auxiliary localization networks, it can be layered onto existing deployed medical LVLMs, including the LLaMA-V2-7B decoder used in this work, without a new training pipeline. The focus on decoding speed matters for clinical settings where latency affects usability.

Future Directions

• Knowledge deficiency remains hard. Med-VCD's improvements on this category are described as modest, and its hallucination score on LLaVA-Med-1.5 (1.39 ± 0.91) still leaves room above zero. The authors observe that mitigating knowledge-level hallucinations requires strategies beyond purely visual enhancement, since methods like PAI that excel at visual hallucinations fail here.
• Reduce variability between backbones. The effects of most mitigation strategies vary between the two Med-LVLM architectures tested, which the authors attribute to the complexity of knowledge-deficiency hallucinations. Broader architecture coverage would test the generality claim further.
• Improve on context misalignment. Most existing methods achieve limited success on linking images to patient history; Med-VCD is the only method showing meaningful improvement on the LLaVA-Med benchmark, leaving headroom for stronger context-aware approaches.
• Extend to more clinical context and modalities. The paper argues current hallucination benchmarks focus on isolated image analysis and neglect broader clinical context such as treatment plans, diagnostic records, and family history — an evaluation gap it only partially closes.

Target Audience

Researchers and engineers working on multimodal clinical AI, medical VQA, and radiology report generation who need hallucination mitigation that deploys without retraining. It is also relevant to practitioners studying inference efficiency in LVLMs, since the sparse contrastive decoding and Sinking Attention Calibration components are general decoding techniques, and to clinical informatics teams evaluating whether a vision-language model is safe enough for imaging workflows. Readers should be comfortable with attention mechanisms and autoregressive decoding; beginners can follow the problem framing and results tables but may need background reading for the optimization formulation.

Authors’ abstract

Large vision-language models (LVLMs) are now central to healthcare applications such as medical visual question answering and imaging report generation. Yet, these models remain vulnerable to hallucination outputs that appear plausible but are in fact incorrect. In the natural image domain, several decoding strategies have been proposed to mitigate hallucinations by reinforcing visual evidence, but most rely on secondary decoding or rollback procedures that substantially slow inference. Moreover, existing solutions are often domain-specific and may introduce misalignment between modalities or between generated and ground-truth content. We introduce Med-VCD, a sparse visual-contrastive decoding method that mitigates hallucinations in medical LVLMs without the time overhead of secondary decoding. Med-VCD incorporates a novel token-sparsification strategy that selects visually informed tokens on the fly, trimming redundancy while retaining critical visual context and thus balancing efficiency with reliability. Evaluations on eight medical datasets, spanning ophthalmology, radiology, and pathology tasks in visual question answering, report generation, and dedicated hallucination benchmarks, show that Med-VCD raises factual accuracy by an average of 13\% and improves hallucination accuracy by 6\% relative to baseline medical LVLMs.