AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs

AI summary

Overview

Research area: Computer vision and multimodal machine learning — specifically the evaluation of visual reasoning in multimodal large language models (MLLMs/VLMs) and image generation models (IGMs).

Technical level: Intermediate. The paper is written accessibly and requires only a working familiarity with vision-language models, text-to-image models, and benchmark evaluation concepts. No deep mathematical background is needed.

Scope: The paper introduces AMVICC, a benchmark that adapts the existing MMVP visual reasoning benchmark into paired image-understanding and image-generation tasks, then uses it to profile where 11 MLLMs and 3 IGMs fail across 9 categories of visual reasoning.

What This Paper Is About

Multimodal models have become highly capable, yet they still fail at visual tasks humans find trivial — confusing left with right, miscounting objects, or misjudging spatial relationships. Prior benchmarks measure these failures separately for vision-language models (which read images) and image generation models (which produce them), but nobody had systematically compared the two. This paper builds a shared benchmark, AMVICC, so that the same visual concept is tested in both directions — image-to-text and text-to-image — and asks whether the two modality families fail for the same underlying reasons.

Key Contributions

1. A new cross-modal benchmark (AMVICC). The authors adapt the 300 questions from the MMVP benchmark into 600 new image-generation prompts — 300 implicit (general scene descriptions) and 300 explicit (prompts specifying the precise visual feature the corresponding MMVP question tests). This allows one visual concept to be probed in both image interpretation and image generation.

2. A 9-category taxonomy for visual reasoning. Every question and prompt is assigned to exactly one of nine categories: Orientation and Direction; Presence of Specific Features; State and Condition; Quantity and Count; Positional and Relational Context; Color and Appearance; Structural and Physical Characteristics; Text; and Viewpoint and Perspective.

3. A large empirical profile of failure modes. Eleven MLLMs (GPT-4o, Gemini 2.5 Pro, Claude Opus 4.1, Claude Sonnet 4, Grok 4, Llama 3.2 90B Vision Instruct, Llama 4 Maverick, Llama 4 Scout, Gemma 3 27B, Qwen2.5 VL 72B Instruct, Pixtral Large 2411) and three IGMs (DALL·E 3, Gemini 2.5 Flash Image, Stable Diffusion 3.5 Large) are evaluated, with failure explicitly defined as individual accuracy below 80% or pair accuracy below 70%.

4. Ablation studies isolating causes of failure. Three follow-up experiments test whether failures come from prompt wording, model randomness, or architecture — concluding that conceptual misunderstanding, not sampling noise, drives IGM errors.

Main Findings

• Shared failure modes exist across modalities. Quantity and Count, Positional and Relational Context, Orientation and Direction, and State and Condition are weak spots for both MLLMs and IGMs. Quantity and Count is the worst category overall for VLMs (individual category average 75.23%, pair 56.06%).

• Text is the sharpest divergence. MLLMs handle textual content in images reasonably well (category average roughly 82%), but all three IGMs fail on Text in both individual and pair explicit accuracy (IGM category averages 64.29% and 38.10%). This is the clearest evidence of modality-specific rather than shared limitation.

• IGMs actually beat MLLMs on viewpoint. In Viewpoint and Perspective, image generators outperform the vision-language models, showing the weaknesses are not simply mirrored.

• Model-specific outliers are real. xAI's Grok 4 is the worst MLLM (60.76% individual average, 39.67% pair average) and is the only model to fail the Color and Appearance category. Meta's Llama 3.2 90B Vision Instruct is the best (91.88% / 84.18%), with no individual failure modes and only one pair failure mode, and it outperforms GPT-4o in every category except Text and Color and Appearance.

• IGMs show a much wider quality spread. Gemini 2.5 Flash Image is far ahead (90.46% individual explicit, 83.08% pair explicit), while Stable Diffusion 3.5 Large fails every category (60.63% / 35.04%), bottoming out at 12.50% pair accuracy in Positional and Relational Context — the lowest score recorded anywhere in the study.

• Pair accuracy is always lower than individual accuracy. Because a pair only counts as correct when both images or both answers are right, the drop is substantial across the board — MLLM category average falls from 80.34% to 64.18%, IGM from 76.14% to 59.16%.

• IGMs struggle with negative and implied instructions. Models routinely include objects that were supposed to be hidden or omitted, and fail to reorient a viewpoint to conceal a specified element (e.g., generating a keyboard with the "Z" key hidden). Sometimes elements meant to be fully hidden are only partially hidden, or vice versa — near-misses that still count as failures.

• Prompt rewording does not help. A linguistic sensitivity ablation using GPT-5 to make 40 prompts more targeted produced no improvement, and actually caused a drop in Pair Explicit accuracy for Presence of Specific Features (from 100% to 0% on one category slice).

• Randomness is not the explanation. Running the same 40 prompts three times through DALL·E 3 yielded stable aggregate accuracies (e.g., pair explicit 80%, 70%, 80%), indicating that failures reflect genuine conceptual misunderstanding rather than sampling variance.

• Architecture and scale are not decisive. Llama 4 Maverick (17B active / 128 experts) and Llama 4 Scout (17B active / 16 experts) perform similarly despite very different total parameter counts, suggesting that parameter scale alone does not drive elementary visual reasoning ability.

Methodology in Plain English

The team started from MMVP, a benchmark that shows a model two visually similar images and asks a question distinguishing them. They took all 300 MMVP questions and assigned each to one of nine visual reasoning categories.

For the vision-language side, they fed each MMVP question and image pair to 11 MLLMs and used GPT-4 to grade the answers as right or wrong. Individual accuracy is the share of questions answered correctly; pair accuracy requires both questions in a matched pair to be correct.

For the image generation side, the authors hand-wrote two prompts per MMVP question. The implicit prompt describes the general scene ("a dog in grass") and tests whether the model can set up the right context at all. The explicit prompt adds the specific detail the MMVP question hinges on ("a dog in grass looking to the right") and tests whether the model can control that precise attribute. Three IGMs generated images from these 600 prompts; human evaluators — with multiple reviewers double-checking — scored each image against a rubric. An implicit image passes if it satisfies the prompt; an explicit image passes only if it contains the specific required feature.

Three ablations followed: rewording 40 prompts to see if better phrasing helps, regenerating the same prompts three times to measure randomness, and comparing two related models with different architectures (Llama 4 Maverick vs. Scout) to separate architecture effects from scale effects.

Why This Matters

Impact on research. The paper provides the first directly comparable failure-mode profile across image understanding and image generation using identical underlying visual concepts. This matters because it lets researchers ask whether a general-purpose visual reasoning deficit exists across modalities, or whether each modality has its own distinct weaknesses. The finding that Text is fine for readers but broken for generators is a concrete example of a modality-specific gap that a single-modality benchmark would have missed.

Real-world applications:

• Content creation and design tools. Advertising, e-commerce, and marketing pipelines depend on generators placing products, text, and logos exactly where prompts specify. Failures in Quantity and Count and Positional and Relational Context translate directly into unusable commercial output.
• Accessibility. Alt-text and image-description systems for blind and low-vision users rely on the same MLLM visual reasoning being tested; the persistent Quantity and Count weakness means descriptions can silently misreport how many objects are present.
• Robotics and embodied agents. Any agent that must interpret a camera feed and act on spatial relationships inherits these orientation, position, and counting errors.
• Medical and scientific imaging. The paper's framing connects to clinical benchmarks like GMAI-MMBench; miscounting or misjudging orientation in a scan is a high-stakes version of the same elementary failures.

Industry relevance. The benchmark covers both open- and closed-source models from OpenAI, Google, Meta, Anthropic, xAI, Mistral, Qwen, and Stability AI, and it reveals large performance spreads within the same modality — Gemini 2.5 Flash Image versus Stable Diffusion 3.5 Large differ by roughly 30 percentage points in overall accuracy. That is directly useful signal for model selection, procurement, and for teams deciding where to invest in capability improvements.

Future Directions

1. Expand the category coverage and scale. The authors explicitly invite extensions of MMVP or AMVICC to a wider range of visual understanding categories and to larger test sets, and to replicate the study with more compute to confirm the results at scale.

2. Determine why the failure patterns diverge. The paper documents where models fail but cannot explain why. In particular, the Text gap between MLLMs and IGMs, and the Viewpoint advantage of IGMs, remain unexplained and are natural targets for mechanistic follow-up.

3. Build a human control group. The evaluation currently has no human baseline for the image generation tasks, so performance can only be interpreted relationally between models. A human comparison would establish how far from achievable these models actually are.

4. Test whether shared failures stem from shared causes. Because many models are trained on structurally similar image-caption data, the authors suggest that shared failure modes might originate in shared training data rather than shared architecture — a hypothesis that would require controlled training experiments to test.

5. Address negative and implied instructions. IGMs consistently mishandle prompts containing "no" or "without," and cannot infer indirect strategies for hiding objects. This is a concrete, addressable capability gap for future text-to-image work.

Target Audience

This paper is most useful for benchmark designers and evaluation researchers working on multimodal systems, practitioners selecting between MLLMs and IGMs for visual tasks, and researchers studying cross-modal alignment who want evidence on whether vision and generation share underlying representations. It is also a good entry point for graduate students or advanced undergraduates looking for a tractable, well-structured benchmark methodology, since the approach — adapting an existing benchmark rather than building one from scratch — is reproducible and clearly documented. Readers focused purely on state-of-the-art image quality or generation aesthetics will find less here; the focus is squarely on elementary reasoning and instruction-following, not fidelity or realism.

Authors’ abstract

We investigate visual reasoning limitations of both multimodal large language models (MLLMs) and image generation models (IGMs) by creating a novel benchmark to systematically compare failure modes across image-to-text and text-to-image tasks, enabling cross-modal evaluation of visual understanding. Despite rapid growth in machine learning, vision language models (VLMs) still fail to understand basic visual concepts such as object orientation, quantity, and spatial relationships, which highlights gaps in elementary visual reasoning. By adapting MMVP benchmark questions into explicit and implicit prompts, we create \textit{AMVICC}, a novel benchmark for profiling failure modes across various modalities. After testing 11 MLLMs and 3 IGMs in 9 categories of visual reasoning, our results show that failure modes are often shared between models and modalities. However, certain failures are model-specific and modality-specific, and this can potentially be attributed to various factors. IGMs consistently struggle to manipulate specific visual components in response to prompts, especially in explicit prompts, suggesting poor control over fine-grained visual attributes. Our findings apply most directly to the evaluation of existing state-of-the-art models on structured visual reasoning tasks. This work lays the foundation for future cross-modal alignment studies, offering a framework to probe whether image generation and visual interpretation failures stem from shared limitations. These insights can guide future improvements in unified vision-language modeling.