Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification

AI summary

Overview

• Research area: Trustworthy multimodal AI — uncertainty quantification and misbehavior detection for large vision-language models (LVLMs).
• Technical level: Intermediate (assumes some familiarity with uncertainty quantification, model output heads, and evaluation metrics such as AUROC/AUPR).
• One-sentence scope: The paper introduces EUQ, a training-free framework that decomposes epistemic uncertainty in LVLMs into conflict (CF) and ignorance (IG) using Dempster-Shafer Theory, and shows it detects four distinct misbehavior categories better than sampling-, probability-, and hidden-state-based baselines.

What This Paper Is About

LVLMs can produce unreliable or harmful outputs—hallucinations, jailbreak-compliant responses, adversarially induced errors, and out-of-distribution (OOD) failures—when given challenging, distribution-shifted, or adversarial inputs. The authors call these misalignments with human expectations "misbehaviors" and argue they stem from two different sources of epistemic uncertainty: internal conflict among the model's own evidence, and ignorance from missing supporting information. Existing uncertainty quantification methods only capture total predictive uncertainty, so they cannot tell these two causes apart; EUQ is proposed to separate them in a single forward pass without any training.

Key Contributions

1. Diagnosis of two uncertainty sources. The paper identifies that diverse LVLM misbehaviors primarily arise from internal contradictions and missing supporting information, and proposes a computationally efficient Dempster-Shafer Theory (DST)-based detection method that captures these fine-grained uncertainties in a single forward pass.
2. Explicit decomposition into CF and IG. EUQ interprets pre-logits features of the LVLM output head as positive (supporting) and negative (opposing) evidence, converts them into basic belief assignments via a closed-form optimal solution under the Least Commitment Principle, and fuses them with Dempster's rule to yield conflict (CF) and ignorance (IG).
3. Layer-wise evidential uncertainty dynamics. A layer-wise analysis of CF and IG across decoder layers provides a new perspective on how internal representations evolve, and shows that certain layers can distinguish among all four misbehavior categories.
4. Extensive evaluation with Misbehavior-Bench. Experiments on four advanced LVLMs over hallucination, jailbreak, adversarial, and OOD scenarios show consistent gains over strong baselines, with reported improvements of 10.4%/7.5% in AUROC and 5.3%/5.5% in AUPR (the abstract cites up to 10.5% relative AUROC improvement). The code and dataset are released.

Main Findings

• EUQ beats strong baselines on average. Across LVLMs and datasets, the aggregate table reports CF at 0.812 AUROC / 0.783 AUPR and IG at 0.783 AUROC / 0.785 AUPR, versus HiddenDetect 0.707 / 0.658, LN-PE 0.704 / 0.660, PE 0.701 / 0.656, SC 0.626 / 0.730, and SE 0.624 / 0.661. CF improves over HiddenDetect by 0.5% AUROC and 0.4% AUPR, even though HiddenDetect was designed specifically for jailbreak detection.
• Hallucinations track conflict; OOD failures track ignorance. Observation 2 states that hallucinations are more easily detected by CF while OOD failures are more effectively captured by IG. On hallucination data, CF reaches average AUROC 0.761 and AUPR 0.824; on OOD inputs, IG reaches average AUROC 0.948 and AUPR 0.846. For jailbreak and adversarial cases, CF and IG perform comparably.
• Layer-wise dynamics are consistent across misbehavior types. Observation 1: across decoder layers, ignorance tends to decrease (deeper layers accumulate more supporting cues) while conflict increases (evidential support becomes increasingly polarized across features). The authors connect this to the information-bottleneck perspective.
• CF and IG separate misbehavior distributions more clearly than entropy. One-vs-rest density comparisons show CF and IG yield cleaner separations than predictive entropy, with adversarial examples the most distinguishable because of the pronounced distributional shift from pixel-level perturbations. The paper notes this separation is currently apparent only in certain decoder layers.
• Models differ widely in raw accuracy, with adversarial and OOD cases hardest. The accuracy table reports hallucination averages of 0.647, jailbreak 0.768, adversarial 0.389, and OOD 0.451 across DeepSeek, Qwen, Intern, and MoF. Adversarial examples produce the lowest accuracy, followed by OOD inputs; jailbreak samples show the highest accuracy.
• Strongest single-cell results cluster in adversarial and OOD settings. IG reaches 0.999 AUROC on OOD for DeepSeek and on adversarial for MoF, and 0.997 on OOD for Qwen; CF reaches 0.996 on OOD for Qwen and 0.921 AUROC / 0.928 AUPR on adversarial for DeepSeek.
• Prompt-based abstention fails. In the hallucination scenario, models almost never selected a "None of the above" option: Table 5 reports Qwen at 0.27% and Intern at 0.00%; after reinforcing the instruction with "If you are unsure, please select 'None of the above'", Table 5 reports 4.93% for Qwen and 0.53% for Intern, indicating persistent overconfidence.
• EUQ is efficient. Table 6 reports per-example runtimes including 9.6×10⁻² s for model inference, 8.9×10⁻¹ s for SC, 9.0×10⁻¹ s for SE, and 9.1×10⁻⁴ s for CF. The text states that sampling-based UQ incurs roughly 10× overhead while EUQ needs only a single forward pass with no specialized layers or auxiliary models.
• Robust to temperature, non-monotonic in model scale. Testing eight temperature settings from 0.1 to 1.4 leaves CF and IG stable. Comparing models with 4B, 8B, 26B, and 38B parameters, detection is strong for the 4B and 38B models: small models make obvious errors that are easy to catch, medium models make subtler errors, and large models are mostly correct so remaining errors are easier to detect.

Methodology in Plain English

The method starts from the linear projection layer that LVLMs use to turn internal features into token predictions. The authors treat the input to that layer (the pre-logits features Z) as evidence for or against each candidate output feature. For every pair of an input feature and an output feature, they compute an "evidence weight" using an element-wise affine transformation, fitting the two parameter matrices in closed form (Lemma 1) under the Least Commitment Principle—a conservative rule that assigns support only where evidence justifies it, and which prevents trivial solutions.

Those evidence weights are split into positive and negative parts, representing support for an output hypothesis and support for its complement. Each part is turned into a simple mass function with a saturating exponential form, and because mass functions sharing the same focal sets combine additively in their evidence weights (Lemma 2), the fusion avoids enumerating the full power set of hypotheses. Dempster's rule then combines positives among themselves and negatives among themselves; CF is computed as the conflict between the combined positive and negative evidence, and IG as the total missing mass. Theorem 1 gives closed-form expressions for both, so no combinatorial explosion occurs. Responses are generated token by token; sentence-level uncertainty is the average of token-level values.

Evaluation spans hallucination (POPE, 1000 multiple-choice items; R-Bench, 1000 multiple-choice items), jailbreak (FigStep, 200 open-ended; Hades, 200 open-ended; VisualAdv, 600 open-ended; typographic attacks following Goh et al., 1800 multiple-choice), adversarial attacks (ANDA, 200 yes-and-no; PGN, 200 yes-and-no), and OOD (Xu et al., 1300 yes-and-no). Four LVLMs are tested: DeepSeek-VL2-Tiny, Qwen2.5-VL-7B, InternVL2.5-8B, and MoF-Models-7B, spanning SwiGLU and MoE architectures. Correctness uses ROUGE-L with a >0.5 threshold for multiple-choice and yes/no tasks, and HarmBench's official classifier for open-ended jailbreak tasks. Detection is scored with AUROC and AUPR. For sampling baselines (SC and SE), 10 responses per question are generated, with temperature 0.1 for the first sample and 1.0 for the rest; all experiments run on NVIDIA H800 PCIe GPUs.

Why This Matters

Impact on research. The paper reframes LVLM misbehavior detection as a problem of distinguishing why a model is uncertain rather than merely how uncertain it is. Because EUQ is training-free and needs only one forward pass over pre-logits features, it offers a cheaper alternative to Bayesian approximations, repeated sampling, and auxiliary-model scoring. The layer-wise CF/IG curves also provide an interpretability handle on how internal representations evolve during decoding.

Real-world applications (as named in the paper):

• Identity authentication.
• Autonomous driving.
• Medical diagnosis.
• Other critical deployments where hallucinated or jailbroken outputs carry high risk.

Industry relevance. The efficiency profile matters for production: sampling-based methods require roughly 10× inference overhead, and prompting models to express or abstain from uncertainty proved unreliable in the paper's tests. A single-pass, no-auxiliary-model detector is more practical for real-time monitoring, guardrails, and routing decisions in deployed multimodal systems, and the released Misbehavior-Bench dataset gives teams a shared benchmark spanning four failure modes.

Future Directions

• Making the layer-wise separation robust. The paper notes that clear separation among misbehaviors is currently apparent only in certain decoder layers; identifying and selecting those layers reliably is an open problem.
• Combining CF and IG into a single detector. CF wins on hallucinations and IG on OOD failures, so a joint or routing scheme that exploits both signals across all four categories is a natural extension.
• Scaling and architecture generality. Scale effects were analyzed at 4B, 8B, 26B, and 38B parameters, with an additional 72B check reported in the appendix; whether the CF/IG observations hold across larger and more architecturally diverse LVLMs remains open.
• Extending beyond the four studied misbehavior categories. The paper's framework is general to the output head, but the evaluation is limited to hallucinations, jailbreaks, adversarial vulnerabilities, and OOD failures, and the discussion section (including the relation to EDL-based methods) is truncated in the provided content, so the authors' own stated future work is not fully reported here.

Target Audience

Researchers and engineers working on trustworthy multimodal AI, uncertainty quantification, and model safety; practitioners building guardrails or monitoring for deployed vision-language systems; and readers interested in applied Dempster-Shafer / evidential reasoning methods who want a concrete, training-free example applied to large models.

Authors’ abstract

%Large vision-language models (LVLMs) have shown substantial advances in multimodal understanding and generation. However, when presented with incompetent or adversarial inputs, they frequently produce unreliable or even harmful content, such as fact hallucinations or dangerous instructions. This misalignment with human expectations, referred to as \emph{misbehaviors} of LVLMs, raises serious concerns for deployment in critical applications. These misbehaviors are found to stem from epistemic uncertainty, specifically either conflicting internal knowledge or the absence of supporting information. However, existing uncertainty quantification methods, which typically capture only overall epistemic uncertainty, have shown limited effectiveness in identifying such issues. To address this gap, we propose Evidential Uncertainty Quantification (EUQ), a fine-grained method that captures both information conflict and ignorance for effective detection of LVLM misbehaviors. In particular, we interpret features from the model output head as either supporting (positive) or opposing (negative) evidence. Leveraging Evidence Theory, we model and aggregate this evidence to quantify internal conflict and knowledge gaps within a single forward pass. %We extensively evaluate our method across four categories of misbehavior, including hallucinations, jailbreaks, adversarial vulnerabilities, and out-of-distribution (OOD) failures, using state-of-the-art LVLMs, and find that EUQ consistently outperforms strong baselines, showing that hallucinations correspond to high internal conflict and OOD failures to high ignorance. Furthermore, layer-wise evidential uncertainty dynamics analysis helps interpret the evolution of internal representations from a new perspective. The source code is available at https://github.com/HT86159/EUQ.