Uncertainty Quantification for Multimodal Large Language Models with Incoherence-adjusted Semantic Volume

AI summary

Overview

• Research area: Uncertainty quantification and trustworthy deployment of Multimodal Large Language Models (MLLMs), spanning vision-language QA, audio-language QA, video-language QA, and image/audio generation.
• Technical level: Intermediate. The paper pairs an implementable, training-free pipeline with supporting theory (determinantal point process style quality-diversity kernels, entropy decompositions, concentration bounds), but the core idea is described geometrically and can be followed without deep background.
• Scope: The paper introduces UMPIRE, a training-free framework that scores how uncertain an MLLM is on a given multimodal task instance by combining the semantic diversity of its sampled responses with the model's own internal incoherence/confidence signals, and shows it outperforms existing uncertainty baselines across image, audio, and video-text benchmarks and non-text generation tasks.

What This Paper Is About

Multimodal Large Language Models can produce fluent, plausible-looking answers that are wrong, which makes them risky to deploy in settings such as medical analysis without a way to flag unreliable outputs. The paper argues that existing uncertainty metrics are limited: they are built for text-only LLMs, tied to one specific modality (typically image-text), dependent on external tools or verifiers, or computationally expensive. The goal is a single training-free uncertainty metric that works across input modalities (image, audio, video) and even non-text output tasks, using only the MLLM's own internal embeddings and conditional probabilities.

Key Contributions

1. A set of stated desiderata for MLLM uncertainty metrics. The authors propose effectiveness criteria — discrimination (R1), risk-score quality with sub-criteria risk-linearity (R2a) and unlabeled normalization to a probability proxy (R2b) — alongside design criteria: multimodal generalizability (R3, with the stricter cross-output-modality version R3'), multimodal coherence (R4), and computational efficiency (R5).
2. UMPIRE, a training-free uncertainty metric. UMPIRE ("Uncertainty using Model Probability Indicators and Response Embeddings") computes the incoherence-adjusted semantic volume of sampled MLLM responses via a DPP-inspired quality-diversity kernel, where "quality" is an incoherence score derived from the model's own response probabilities and "diversity" is measured in the model's embedding space.
3. Theoretical analysis of the metric's structure. The paper shows an exact decomposition of the metric into an unadjusted semantic volume term plus a Monte Carlo estimate of quadratic entropy, explains how the two terms interact, and gives concentration results (sub-Gaussian concentration at O(1/√k), with misranking probability decaying exponentially in kΔ²).
4. Broad empirical evaluation. UMPIRE is tested on image-text, audio-text, and video-text QA benchmarks (including adversarial and out-of-distribution datasets), on image and audio generation, and in a blackbox API setting using a smaller whitebox proxy model — with accompanying ablations, statistical significance tests, and computational-efficiency comparisons.

Main Findings

• Discrimination (R1): UMPIRE achieves the best or second-best AUROC across all baselines and modalities, reporting an average AUROC of 0.81 on image-text datasets (Table 1 average for image datasets: 81.1; average across all modalities: 81.2). It is described as performing especially well on challenging datasets such as OKVQA and AdVQA, where the image-specific Neighborhood Consistency (NC) baseline struggles due to adversarial and out-of-distribution scenarios.
• Risk-linearity (R2a): UMPIRE reaches an average Calibration Pearson Correlation (CPC) of roughly 0.90 across all modality tasks, described as more than 11% higher than the next best metric, and as more stable across tasks than baselines whose CPC fluctuates greatly (average CPC across all modalities: NC not applicable, LN-Ent 76.4, Sem.Ent 72.1, Eigen 81.2, UMPIRE 90.4).
• Calibration (R2b): Using an unlabeled development set of 5% of each dataset to min-max scale the score, UMPIRE achieves low Expected Calibration Error on almost all datasets, with an average of 0.062 (average ECE across all modalities: LN-Ent 0.080, Sem.Ent 0.211, Eigen 0.227, UMPIRE 0.062).
• Multimodal generalizability (R3): Without modality-specific modifications or external tools, UMPIRE performs well on image-text, audio-text (SLUE, SpokenSQ.) and video-text (VidMME) tasks. NC cannot be applied to non-image modalities at all; LN-Ent, Sem.Ent and Eigen can be adapted but consistently perform worse.
• Non-text output (R3'): On image generation (MS-COCO caption data with AnyGPT and NExT-GPT) and audio generation (AudioCaps with NExT-GPT), UMPIRE achieves the highest Pearson correlation between its uncertainty scores and the negative of CLIP/CLAP quality scores. Example: on AnyGPT image generation, correlation was 81.5 ±3.4 for UMPIRE versus 44.0 ±2.8 (PUNC), 16.6 ±9.8 (LN-Ent) and 33.0 ±1.6 (Eigen). Sem.Ent could not be applied because it requires text-specific external processing.
• Multimodal coherence (R4): When image input is progressively corrupted with noise, replaced by a black image, or removed entirely, UMPIRE's performance degrades in the expected order. LN-Ent shows large but inconsistent degradation (for example, removing the image degrades less than using a noisy image), and Eigen is nearly invariant to image removal, indicating it measures response dispersion without multimodal coherence.
• Computational efficiency (R5): NC and Sem.Ent require up to 1000× more compute overhead time than MLLM sampling cost, at roughly 9 s per sample versus roughly 8e-4 s per sample for UMPIRE.
• Selective answering: In a routing scenario where the most uncertain instances are escalated, UMPIRE achieves the highest Area Under the Rejection-Accuracy Curve (AURAC) across all datasets, with an average across all modalities of 70.0 versus 69.3 (Eigen), 67.7 (Sem.Ent), 66.0 (LN-Ent), and NC not applicable outside image tasks.
• Blackbox models: UMPIRE can be applied to closed API models by using a smaller whitebox proxy MLLM to produce embeddings and probabilities. It is reported to consistently and significantly outperform baselines when assessing GPT4o's uncertainty on VQAv2, OKVQA and AdVQA using Llava-v1.5-13b as the proxy, including against the Verbalized Confidence baseline; similar results are reported for other blackbox models (Claude, GPT4o Mini) and other proxy models.
• Component analysis: The semantic volume term and the incoherence term are complementary. The incoherence term is often a strong discriminator but cannot distinguish semantic ambiguity from predominantly lexical variation; the semantic volume term resolves this, improving reliability-curve linearity, CPC and ECE while preserving competitive AUROC. Likelihood Ratio Tests are reported as showing the statistical significance of the volume term in complementing the incoherence term.
• Sampling budget: UMPIRE outperforms baselines across sampling budgets and gives significant margins over single-sample methods even at k = 5.
• Scope limitation stated by the authors: The framework targets confabulations — fluent but arbitrary wrong responses from lack of relevant knowledge. It does not address cases where an MLLM consistently produces the same wrong response when sampled, such as errors from training problems or erroneous data.

Methodology in Plain English

UMPIRE works in four steps and requires no training, no fine-tuning, and no external verifier or reward model.

1. Sample multiple answers. For a task instance, the model generates k responses using normal inference sampling settings, such as temperature T = 1 with nucleus sampling. The paper's theory assumes responses are independent and identically distributed from the model's conditional distribution.
2. Embed each answer. Each response is represented by the normalized last-layer MLLM embedding of its end-of-sequence token, giving a k × d matrix where d is larger than k. Because embeddings are normalized, the samples sit on a hypersphere, and the angles between them capture semantic distance.
3. Score incoherence. For each response, an incoherence score is computed as exp(α(1 − p_i)), where p_i is the model's own probability for that response given the multimodal query and α is a scale constant. Intuitively this is the model's internal doubt about a response, and because the probability is conditioned on all input modalities, it gives multimodal grounding without an external checker.
4. Compute the incoherence-adjusted semantic volume. The embedding Gram matrix is scaled by the incoherence scores, and the uncertainty score is the sample-count-normalized log determinant of the resulting matrix (Equation 4). A more numerically stable implementation uses the equivalent decomposition (Equation 5): a semantic volume term plus an α-weighted average of the incoherence scores. The log determinant is computed via Cholesky decomposition, with complexity O(k³), which the paper notes is negligible next to the cost of MLLM inference at the modest k used.

The single hyperparameter α is set per task without labeled data by a heuristic: the ratio of the median semantic-volume term to the median incoherence term over an unlabeled subset of instances. A small labeled development set would allow tuning that can further improve results. For long generations where response probabilities become very small, the authors suggest heuristics such as length-normalizing log probabilities or considering only a limited answer span.

Baselines compared: Neighborhood Consistency (modality-specific to images), LN-Entropy, Semantic Entropy, Eigenscore (text-only LLM metrics adapted to multimodal input), PUNC (image-generation-specific), and Verbalized Confidence (for blackbox API models). Models used: Llava-v1.5-13b for image-text, Phi-4-multimodal-instruct for audio-text, LLaVA-NeXT-Video-7b-hf for video-text, and NExT-GPT and AnyGPT for non-text generation.

Why This Matters

The paper targets a practical bottleneck in deploying multimodal models: knowing when to trust an answer. A metric that reliably flags unreliable instances enables escalation of hard cases to human experts or larger, more capable models, rather than accepting every output at face value. Its distinctive combination is that it needs no external tools, no modality-specific engineering, and only signals the model already produces, which is what allows a single framework to span image, audio and video input and to be applied to closed API models through a small proxy.

• High-stakes expert assistance: the paper motivates uncertainty quantification for settings such as medical analysis, where reliably flagging uncertain cases before they reach a clinician matters. VQA-RAD appears among the image QA benchmarks evaluated.
• Selective answering and escalation routing: a user with a small local MLLM and a limited budget can route the most uncertain instances to a stronger model or a human, which the paper evaluates with AURAC.
• Robustness screening in adversarial and out-of-distribution use: the AdVQA results indicate the metric helps surface cases where models fail under adversarial or shifted inputs.
• Quality triage for generative media: the image and audio generation experiments suggest the same scoring can indicate whether a task instance is likely to yield a low-quality generated image or audio clip.

Industry relevance. Providers running MLLMs behind closed APIs can adopt UMPIRE via a smaller whitebox proxy model, which the paper explicitly demonstrates with GPT4o. Because the metric adds no external tool calls and has negligible overhead relative to inference (roughly 8e-4 s per sample versus roughly 9 s per sample for the costly baselines), it is compatible with constrained inference pipelines — the paper frames this as condition (b) of its efficiency desideratum, relaxed only to permit a small, cheap proxy model for blackbox systems.

Future Directions

• Uncertainty for consistent errors, not just confabulations. The authors state they do not consider cases where the MLLM repeatedly produces the same wrong answer, such as errors stemming from training problems or erroneous data. Extending coverage to those failure modes is an open problem.
• Strengthening the blackbox approach. The proxy-model method relies on the assumption that performant MLLMs share sufficiently similar multimodal features and that UMPIRE is robust to distortions in the embeddings and probabilities it receives. Testing this assumption across a wider range of proxy and target model pairs is a natural next step.
• Scaling the non-text output direction. The paper describes cross-output-modality generalizability (R3') as not the focus of the work and only provides some analysis, so systematic evaluation of image and audio generation across models and datasets remains open.
• Handling long generations. The authors note that response probabilities become small for long outputs and suggest heuristics such as length normalization or restricting to a limited answer span; principled treatment of this regime is unresolved.
• Tuning the single hyperparameter. The paper uses a label-free heuristic for α and notes that a small labeled development set can further boost results, leaving the question of how best to set or learn α in deployment.

Target Audience

Researchers and engineers working on uncertainty quantification, hallucination and confabulation detection, and reliable deployment of multimodal LLMs. It is also relevant to practitioners building escalation or human-in-the-loop review pipelines, to teams serving closed API models who need uncertainty estimates without model internals, and to readers interested in applying determinantal point process style quality-diversity ideas to language model outputs. The mix of a practical recipe, a decomposition-based analysis, and cross-modality benchmarks makes it useful both for those who want to implement the metric and for those studying the theory of uncertainty scores.

Authors’ abstract

Despite their capabilities, Multimodal Large Language Models (MLLMs) may produce plausible but erroneous outputs, hindering reliable deployment. Accurate uncertainty metrics could enable escalation of unreliable queries to human experts or larger models for improved performance. However, existing uncertainty metrics have practical constraints, such as being designed only for specific modalities, reliant on external tools, or computationally expensive. We introduce UMPIRE, a training-free uncertainty quantification framework for MLLMs that works efficiently across various input and output modalities without external tools, relying only on the models' own internal modality features. UMPIRE computes the incoherence-adjusted semantic volume of sampled MLLM responses for a given task instance, effectively capturing both the global semantic diversity of samples and the local incoherence of responses based on internal model confidence. We propose uncertainty desiderata for MLLMs and provide theoretical analysis motivating UMPIRE's design. Extensive experiments show that UMPIRE consistently outperforms baseline metrics in error detection and uncertainty calibration across image, audio, and video-text benchmarks, including adversarial and out-of-distribution settings. We also demonstrate UMPIRE's generalization to non-text output tasks, including image and audio generation.