Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration

AI summary

Overview

Research area: AI evaluation and benchmark quality analysis for vision-language models (multimodal benchmarks), combining psychometrics (item response theory) with information-theoretic analysis of item structure.

Technical level: Advanced. The paper assumes familiarity with item response theory, partial information decomposition, and vision-language model evaluation practice.

Scope: A framework (SRCoD) that diagnoses individual multimodal benchmark items by jointly using the internal evidence structure of each item and the observed responses of a pool of vision-language models under controlled modality conditions.

What This Paper Is About

Multimodal benchmarks are supposed to test whether a model can reason across images and text together, but many items can be answered from one modality alone or from answer-option patterns, so they do not actually test cross-modal reasoning. Existing ways of finding these shortcut items look either only at the item's internal content structure or only at how a chosen pool of models responds to it, and neither view alone gives a reliable item-level diagnosis. The paper builds a method that combines both signals to label each item as redundant, visual-dominant, textual-dominant, or synergistic, and validates those labels against human annotators.

Key Contributions

1. A structure-response co-calibrated view of benchmark diagnosis. The authors argue that intrinsic item structure tells you what evidence an item contains, while extrinsic model responses tell you how the item functions in evaluation, and that reliable diagnosis needs both. They instantiate this as SRCoD, a structure-anchored multimodal IRT framework.

2. An interpretable, item-level diagnostic profile. SRCoD produces a four-coordinate profile over redundant, visual-dominant, textual-dominant, and synergistic evidence, so each diagnosis comes with an attribution of which evidence source explains the item's behavior rather than just an aggregate accuracy number.

3. A controlled human-labeled validation protocol. The authors build a diagnostic validation set of 1,000 items per benchmark across ScienceQA, MMMU, and SEED-Bench, where annotators judge whether an item is solvable from visual input alone, textual input alone, either modality, or only their combination — providing item-level ground truth that prior intrinsic-structure work did not collect.

4. Evaluation under two complementary protocols. SRCoD is validated both by benchmark-refinement oriented validation (does a selected compact subset preserve the model ranking of the full clean benchmark?) and by human-labeled item validation, outperforming intrinsic-structure and extrinsic-response baselines in both.

Main Findings

• Best human-labeled diagnostic accuracy: SRCoD achieves the best average accuracy (0.613) and macro-F1 (0.600) across the three benchmarks, versus 0.558/0.542 for the strongest baseline M3IRT and 0.250/0.239 for Random. Per benchmark, SRCoD reaches 0.607/0.588 on ScienceQA, 0.548/0.538 on MMMU, and 0.685/0.673 on SEED-Bench.

• Structure-only signals are insufficient: The LSMI baseline, which uses only intrinsic item structure, scored 0.346 average accuracy but only 0.205 average macro-F1. Its macro-F1 fell below the Random baseline on MMMU (0.171 vs 0.241) and SEED-Bench (0.189 vs 0.242), showing that structural signals alone do not give balanced diagnosis across all four categories.

• Response-only signals are useful but not enough: Extrinsic-response baselines (IRT, MIRT, M3IRT) generally beat the structure-only baseline, but all trail SRCoD, which the authors attribute to response evidence alone not resolving fine-grained modality dependence. Notably, the simple Raw baseline reached 0.503 accuracy on MMMU — above IRT (0.384), MIRT (0.371) and M3IRT (0.486) — but its macro-F1 (0.435) stayed below M3IRT's (0.484).

• Ranking preservation under contaminated pools: In the benchmark-refinement protocol, where candidate pools are 50% clean items and 50% synthetic low-quality items made by swapping images or text across questions, SRCoD maintains high Spearman rank correlation across subset sizes from 5% to 25%. It is described as competitive with M3IRT on ScienceQA, best on MMMU at subset sizes of 10% and above, and near-ceiling from 10% onward on SEED-Bench. Exact correlation values are shown only in Figure 2 and are not reported numerically in the text.

• Both components matter (ablation): Removing the entropy-based confidence weight (the "w/o entropy" variant) degrades accuracy and macro-F1, and removing the profile-level structural alignment loss (the "w/o struct." variant) also reduces performance, across all three benchmarks. The authors conclude that structural signals help most when they both anchor item parameters and regularize the final profile.

• Stability under small model pools: When the diagnostic model pool is randomly subsampled to n ∈ {4, 6, 8, 10, 12} models over 10 trials, SRCoD retains more of its full-pool accuracy and macro-F1 than M3IRT, particularly at 4 and 6 models, and shows smaller trial-to-trial variation.

• A concrete diagnostic example: For an item asking for the capital of Wyoming with a map highlighting the state, SRCoD labels it Redundant, because the question text alone identifies the state and the highlighted region does so visually.

Methodology in Plain English

The method diagnoses each benchmark item in four steps.

Step 1 — Describe how evidence is spread inside the item. Using a sample-wise estimator called LSMI, which builds on partial information decomposition, the authors compute for each item a four-number summary of its multimodal interaction: how much answer-relevant information is redundant across image and text, how much is unique to the image, how much is unique to text, and how much only appears when both are combined. These numbers are converted into a probability-like "structural prior" over the four diagnostic categories.

Step 2 — Account for how confident that prior is. Some items yield vague structural estimates. SRCoD assigns each item a weight based on the entropy of its structural prior, so near-uniform (ambiguous) priors count for little and confident ones count for more.

Step 3 — Fit a response model anchored to that prior. The authors collect yes/no correctness from a pool of vision-language models under four modality conditions (no modality, visual only, textual only, both), and fit a modality-conditioned IRT model in which each model has a four-dimensional ability vector and each item has four-dimensional discrimination and difficulty vectors. Crucially, the item parameters are not learned freely: they are parameterized through a projection of the item's structural interaction vector plus learned residuals, so structure sets the starting point and observed responses can revise it.

Step 4 — Read out a calibrated profile. After fitting, the model predicts correctness for each item under each modality condition at the pool's average ability. Differences between these predicted probabilities (shared gain over the no-modality baseline, asymmetry between visual and textual, and extra gain from both modalities together) become the final four-coordinate profile after a softmax.

Training combines a standard response likelihood term with a weighted KL-divergence term pulling the calibrated profile toward the structural prior.

Why This Matters

Impact on research. The paper reframes benchmark diagnosis as a calibration problem between what an item contains and how models behave on it, rather than a choice between content analysis and response analysis. It also contributes an evaluation resource: a controlled, human-annotated, four-category evidence-sufficiency label set of 1,000 items per benchmark, which prior intrinsic-structure work lacked and which prior extrinsic-response work approximated only through ranking-preservation proxies.

Real-world applications:

• Benchmark auditing and curation: identifying shortcut items in existing multimodal benchmarks so they can be removed, repaired, or reported separately from genuinely cross-modal items.
• Training and evaluation data selection: choosing compact, high-quality evaluation subsets that preserve the conclusions of a full benchmark, which reduces evaluation cost for model developers.
• Model procurement and comparison: giving buyers of vision-language models a way to check that the benchmark scores they rely on reflect multimodal reasoning rather than language priors.
• Diagnosing failure modes in deployed systems: knowing whether a task family is redundantly solvable informs whether a system needs an image pipeline at all, or whether text alone suffices.

Industry relevance. Model and dataset teams routinely pay for large multimodal evaluation runs and for human annotation. SRCoD's robustness under small diagnostic model pools (tested down to 4 models) matters because collecting modality-conditioned responses from many commercial and open-weight models is expensive. Its four-way output is also directly actionable compared with a single quality score.

Future Directions

• Beyond multiple-choice vision-language benchmarks: the authors explicitly note SRCoD has not been validated on open-ended generation tasks, interactive multimodal evaluation, or modalities beyond vision and language.
• Reducing dependence on the diagnostic model pool: the paper acknowledges that profiles remain sensitive to which models are included, giving examples where limited understanding of molecular diagrams may overestimate cross-modal synergy and difficulties with contingency tables may underestimate visual contribution. Better pool design or pool-robust estimation is an open problem.
• Graded rather than binary evidence-sufficiency labels: the authors propose finer annotations than the current single-modality-solvable judgments, which can depend on annotator background knowledge and item ambiguity.
• Improving the structural prior itself: sample-wise interaction estimates are approximations affected by representation quality, so better estimators would likely tighten the anchoring signal.

Target Audience

Researchers and practitioners working on multimodal model evaluation, benchmark construction, and dataset quality auditing, especially those familiar with or willing to learn item response theory. It is also relevant to psychometricians applying IRT to machine evaluation, and to evaluation engineers at model labs or third-party benchmarking organizations who need to justify which items their reported scores rest on. Readers looking for a ready-to-use benchmark-cleaning tool should note the paper's own caveat that SRCoD produces estimates of modality dependence rather than definitive judgments of human evidence sufficiency.

Authors’ abstract

Multimodal benchmarks are central to evaluating whether models can reason over information from multiple modalities. However, existing literature has revealed that some benchmarks may contain shortcut items that can be answered from a single modality or from answer-option regularities alone, thus failing to reflect models' true cross-modal inference ability. Existing methods for assessing benchmarks typically rely either on intrinsic item structure or extrinsic model response behavior, which are not adequate for a comprehensive diagnosis. To fill in this gap, we propose SRCoD, a structure-response co-calibrated diagnosis framework for item-level diagnosis of modality dependence in multimodal benchmarks. SRCoD estimates sample-wise multimodal information structure from benchmark content and target answers, and uses this signal as a soft structural anchor for modality-conditioned item response modeling. The learned model produces a calibrated item-level diagnostic profile that reflects how strongly an item depends on different forms of modality evidence, enabling fine-grained benchmark analysis beyond aggregate accuracy. SRCoD also provides interpretable attribution of the diagnosis results. Extensive experiments under two complementary protocols, i.e., benchmark-refinement oriented validation and human-labeled item validation, demonstrate that SRCoD outperforms state-of-the-art benchmark diagnosis methods.