DiaVLo: Diagnosing Behaviours of Vision-Language Models

AI summary

Overview

Research area: Evaluation and diagnosis of vision-language models (VLMs), sitting at the intersection of multimodal benchmarking, explainable AI, self-explanation, and causal inference.

Technical level: Advanced. The paper assumes familiarity with VLM architectures (visual encoder, projection layer, decoder-only LLM), scene graph generation, double machine learning, and explanation evaluation.

Scope: The paper introduces DiaVLo, a diagnostic framework that builds specifications of desired and observed VLM behaviours, classifies their alignment, and estimates which visual concepts causally influence model outputs, evaluated on four open-source VLMs across four visual question answering datasets.

What This Paper Is About

Current VLM evaluation (benchmarks) tells you how well a model performs a task, but not which behaviours it actually exhibits, nor whether those behaviours match what a human would consider correct and relevant. DiaVLo addresses this gap by pairing human-curated "SHOULD-KNOW" specifications of desired behaviour with VLM-generated "REALLY-KNOW" specifications of observed behaviour, then semantically comparing the two to classify behaviours as aligned, expanded, or divergent, and using causal modelling to quantify concept influence.

Key Contributions

1. A formalisation of the VLM diagnosis problem, defining desired and observed behaviours as triplet-based specifications over concepts and relations, and introducing three behaviour types (Aligned, Expanded, Divergent).
2. The DiaVLo framework, which combines scene graph generation with human curation to build SHOULD-KNOW specifications, elicits REALLY-KNOW specifications from VLM self-explanations, and classifies behaviours through lexical and semantic matching with flexible thresholds.
3. A causal modelling layer that generates counterfactual images via concept-level occlusion and fits a Double Machine Learning estimator (using gradient boosted trees) to estimate concept-level causal effects on VLM outputs.
4. An extensive empirical analysis of four well-known VLMs (InternVL2 8B, LLaVa-1.6 7B, Qwen2.5-VL 7B, ShareGPT4V 7B) across four public datasets in both classification and generation settings, including crowdsourced annotation of SHOULD-KNOW specifications by 520 workers.

Main Findings

• Behaviour labels track performance: SHOULD-KNOW and REALLY-KNOW similarities from DiaVLo are good indicators of VLM performance, measured with bootstrapped Mutual Information between similarity and accuracy (Table 3). For LLaVa-Bench and MMBench, performance is continuous in [0, 1] and MI is theoretically unbounded; for SEED-Bench 2 and VQA v2, performance is discrete in {0, 1} and MI ≤ ln(2) ≈ 0.693.

• Model-wise variation in behaviour-performance coupling: InternVL2 (e.g., MI 3.367 on LLaVa-Bench, 3.200 on MMBench) and LLaVa-1.6 (2.389 and 2.512) showed behaviours more closely related to their performance. Qwen2.5-VL behaved consistently only on LLaVa-Bench (3.008), falling off elsewhere (1.561 on MMBench, 0.188 on SEED-Bench 2, 0.147 on VQA v2). ShareGPT4V showed better consistency on open-ended data (0.810 accuracy with MI 0.558 on VQA v2, but flagged with a very small sample size).

• Most behaviours are not aligned: Across the tested VLMs, DiaVLo classified 6.4% of behaviours as Aligned, 52.5% as Expanded, and 41.1% as Divergent.

• VLMs describe rather than compose: Expanded behaviours revealed that REALLY-KNOW specifications are largely descriptive (properties of objects) rather than compositional (spatial locations of objects), a pattern the authors note aligns with prior research on VLM blind spots and gaps in visual capabilities.

• Different concept organisation from humans: Where similar or identical concepts appeared in both SHOULD-KNOW and REALLY-KNOW sets, the relationships and concept combinations differed, hinting at concept-level latent prioritisation by VLMs that does not align with human organisation under the same input.

• Divergent behaviours carry more reported concepts: Divergent cases involved larger numbers of REALLY-KNOW concepts, which the authors suggest may indicate model uncertainty and attempts to make outputs self-consistent.

• Causal effects concentrate near zero with selected outliers: Concept-level causal estimates generally concentrated around 0.0. Estimates on open-ended datasets spanned multiple concepts and dispersed across output tokens, while multiple-choice datasets showed a higher concentration of zero-valued estimates, possibly indicating VLMs activate on concepts irrelevant to specific questions.

• SHOULD-KNOW curation is largely reliable: A maximum of approximately 11% of behaviours required corrections for trivial mistakes (mostly typos and incorrectly formatted data). Human validation rates were 51.7% (LLaVa-Bench), 58.2% (MMBench), 58.4% (SEED-Bench 2), and 58.5% (VQA v2).

• REALLY-KNOW extraction succeeded for 1754/1996 initial samples, losing approximately 12% due to empty responses and unstable instruction-following, with ShareGPT4V as the loss leader.

• Threshold sensitivity: τ_a plateaued in [0.65, 0.8], while τ_e ≥ 0.4 led to an increasing number of incorrectly classified Divergent samples. With τ_a ≥ 0.7, 22.31% of inspected pairs had only one similar concept and were falsely classified as aligned; a more conservative τ_a ≥ 0.75 reduced this to 17.70% and increased average similarity from 0.77 to 0.84. Setting τ_d < 0.35 already yielded 92.69% of pairs with dissimilar or distinct concepts and relations (mean similarity 0.21). Final settings: τ_a ≥ 0.75, τ_e ≥ 0.35, τ_d < 0.35.

• Causal graph assumptions partly held: Using REALLY-KNOW specifications as causal graphs, structural refutation tests gave mixed results. REALLY-KNOW specifications were not falsified when sufficient counterfactual samples could be produced (approximately 100 or more), and passed baseline permutation tests; otherwise results were considered likely inconclusive due to low test power. Placebo refutation tests on the estimation process were also inconclusive.

Methodology in Plain English

DiaVLo runs two parallel processes and compares their outputs.

Building the specification of desired behaviour ("SHOULD-KNOW"): For each image, the researchers run scene graph generation with IETrans (which uses 70k distinct concept labels and 1.8k relationship labels) to get objects with bounding boxes and relational triplets such as (person, riding, surfboard). Because these descriptions are task-agnostic and can be imprecise, crowdworkers perform two tasks: a verification task (checking correctness and relevance of triplets to the image and prompt, with auto-complete assistance from IETrans labels) and an expansion task (adding missing triplets, with labels and bounding boxes).

Extracting the specification of observed behaviour ("REALLY-KNOW"): Each VLM answers the image-text pair, is then prompted to produce rationales for its answer, and finally is asked to format those rationales into triplets. Bounding boxes for these concepts come from matching against SHOULD-KNOW triplets, falling back to the open-vocabulary detector OWLv2 and similarity matching.

Causal modelling: To distinguish concepts a VLM uses from those it merely sees, the researchers generate counterfactual inputs by occluding combinations of concepts taken from the powerset of the REALLY-KNOW concepts, keeping the text input constant, and re-running inference. A Double Machine Learning estimator then fits two models — one approximating the outcome given covariates, one approximating the treatment given covariates — and computes concept effects from the resulting residuals, with estimates averaged across cross-fitting folds.

Classification: SHOULD-KNOW and REALLY-KNOW specifications are embedded and compared using cosine similarity, with iterative best-match pairing. Behaviours are labelled Aligned (if |RK| ≤ |SK| and similarity ≥ τ_a), Expanded (if |RK| > |SK| and similarity ≥ τ_e), or Divergent (similarity < τ_d).

Informativeness analysis: Mutual Information between similarity and measured performance is bootstrapped, and concept-level causal effect distributions are compared across model pairs, reporting median estimates over n = 100 random data splits.

Why This Matters

Impact on research: The paper reframes VLM evaluation as diagnosis — verifying desired output regularities rather than only aggregating task accuracy — and argues that benchmarking alone does not uncover specific behaviours or study information transfer across modalities. It provides a bridge between interpretability methods and human-facing behavioural specifications, and offers an operational way to check whether a model's stated rationales are consistent with causal influence estimates.

Real-world applications (grounded in the paper's framing):

• Verifying, before deployment, that a VLM exhibits desired behaviours and avoids harmful ones, which the authors state is central to reliable deployment.
• Informing mitigation techniques for curbing unwanted behaviours, such as concept-level prioritisation patterns that diverge from human expectations.
• Supporting red teaming and vulnerability disclosure practices, which the authors suggest bridging with behavioural analysis to reduce misuse risk.
• Providing context to standard evaluation metrics, since behaviour labels correlate with measured performance and may explain why a model scores as it does.

Industry relevance: Practitioners deploying VLMs in multimodal pipelines can use behaviour classification to detect when a model relies on concepts irrelevant to a user request, or when it substitutes broad descriptions for compositional detail — both relevant to quality assurance and to model selection. The framework is model-agnostic in its SHOULD-KNOW construction but produces per-model, per-dataset REALLY-KNOW specifications, matching how organisations evaluate candidate models against their own requirements.

Future Directions

• Replace or augment human curation with LLM judges. The authors explicitly leave the study of LLM judges for diagnosis as future work, noting that LLM judges often require human-written references and can exhibit documented preferential biases. They sketch a Find-Fix-Verify workflow with a Map-Reduce pattern to restrict human effort to a subset of SHOULD-KNOW specifications.

• Improve the faithfulness of self-explanations. The authors state that causal modelling is not equivalent to assessing faithfulness, and suggest factored decomposition as a possible mitigation, while noting that such approaches may still provide limited faithfulness for VLMs.

• Test larger and closed-source models. The experiments covered only 7B–8B models, and the authors acknowledge that applying the framework to larger models may yield different conclusions. Closed-source models were excluded to preserve reproducibility.

• Fully characterise REALLY-KNOW specifications as causal graphs. Refutation tests were inconclusive in low-power regimes. The authors state that definitive conclusions will likely require triangulation with multimodal interpretability research, and that finer-grained masking approaches could alter counterfactual responses and causal estimates.

• Address unstable instruction-following. The tested VLMs sometimes returned empty strings and struggled to follow formatting templates, even across greedy, beam search, and sampling decoding; the authors suspect this stems from predominantly conversational training data and model size.

Target Audience

VLM researchers and evaluation engineers who want more than leaderboard accuracy; explainable AI and causal inference researchers interested in applying counterfactual and DML methods to multimodal models; safety, red-teaming, and model governance practitioners who need behavioural specifications rather than aggregate scores; and HCI or crowdsourcing researchers, given the framework's reliance on human curation tasks and the authors' proposed Find-Fix-Verify pipeline. Readers should be comfortable with VLM architecture terminology, semantic similarity, and the basics of causal effect estimation.

Authors’ abstract

Vision-language models (VLMs) rely on storing and transferring appropriate information across their sub-components. Verifying that the VLMs exhibit desired behaviours, while avoiding harmful ones, is central to their reliable deployment. Yet, methods that identify VLM behaviours remain scarce. We present DiaVLo, a diagnostic framework that leverages human curation and VLMs' generation capabilities to construct specifications of desired and observed VLM behaviours, surfacing potential misalignments. Beyond this, DiaVLo also provides causal estimates to identify the most influential concepts steering VLM behaviours. We evaluate DiaVLo on several open-source VLMs under both classification and generation conditions. Our experiments show that DiaVLo produces behaviour labels that correlate with model performance and provide context for measured performance. DiaVLo surfaced behaviours that are clearly aligned and misaligned, alongside patterns in how VLMs perceive, organise, and prioritise concepts.