Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark

AI summary

Overview

• Research area: Evaluation of unified multimodal models — systems that both understand images and generate them within one architecture.
• Technical level: Intermediate. The paper is readable without deep math, but it assumes familiarity with vision-language models, image generation/editing models, and standard benchmark terminology.
• Scope: The paper introduces an 885-instance, eight-task benchmark (arXiv:2510.13759v3 [cs.CV], from Shanghai Artificial Intelligence Laboratory, University of Science and Technology of China, S-Lab at Nanyang Technological University, and The Chinese University of Hong Kong) that measures whether image generation and image understanding actually help each other, scoring both intermediate steps and final answers.

What This Paper Is About

Unified multimodal models are supposed to combine visual understanding with image generation, but existing benchmarks usually test those two abilities separately, or test them together only superficially. The authors argue that the interesting question is when and how generation and understanding reinforce one another — as when a student draws auxiliary lines to solve a geometry problem, or an artist uses knowledge of optics to paint a scene. Uni-MMMU is built to force that dependency: every task requires a model to either use understanding to guide precise image synthesis, or use generated images as a scaffold for reasoning, with deterministic correct answers so the whole process can be scored automatically.

Key Contributions

1. A bidirectionally coupled benchmark. Eight reasoning-centric tasks across science (physics, chemistry, biology), coding, mathematics, and puzzles, organized into two paradigms — "Generation aids Understanding" (Maze, Sliding Puzzle, Geometry, Jigsaw; 59.2% of the 885 instances) and "Understanding aids Generation" (Science; Code Rendering; 40.8%).
2. A deterministic, dual-level evaluation protocol. Every task has a unique ground-truth answer and verifiable intermediate steps, enabling separate scoring of intermediate visual states and final textual outcomes through programmatic parsers, perceptual metrics, and model-as-a-judge evaluation.
3. Comprehensive multi-discipline evaluation. Six unified models (Bagel, OmniGen2, Ovis-U1, Qwen-Image-Edit, nano-banana, and GPT4.1 + GPT-image) plus six specialized models (generation-only: FLUX.1-Kontext, Imagen 3-001, Imagen 4; understanding-only: Qwen2.5-VL-72B, GPT4.1, Gemini-2.5 Pro) are compared, diagnosing where the synergy holds and where it breaks.
4. A validated judge pipeline with open releases. The authors report agreement between their evaluator and human annotators, and release code, datasets, evaluation tools, and judge configurations at https://github.com/uni-mmmu/Uni-MMMU.

Main Findings

• Generation and understanding reinforce each other most when the dependency is logically strict. The ablation (Table 4) shows that coupling the understanding module with self-generated intermediate states beats running understanding and generation in isolation, and that even imperfect intermediates help. Ground-truth intermediates bring consistent, sizeable gains: Jigsaw rises from 78.0 (GPT-4.1 alone) to 80.0 (GPT) to 98.0 (GPT with ground-truth intermediates); Maze sample-level accuracy goes from 9.4 to 18.1 to 24.8; Sliding from 2.4 to 2.4 to 3.6; the Math column from 16.4 to 17.1 to 27.8; and Science from 61.8 to 84.0 with ground-truth intermediates. The effect is especially pronounced for Maze solution accuracy and Science image correctness.

• There is a clear imbalance: understanding is stronger than generation. Across Table 2, image-generation scores generally trail understanding scores, making generation the primary bottleneck. Average scores for unified models are 44.1 (GPT4.1 + GPT-image), 37.3 (nano-banana), 26.3 (Qwen-Image-Edit), 22.0 (Bagel), 16.5 (Ovis-U1), and 16.0 (OmniGen2).

• Better image generation correlates with better reasoning on scaffolded tasks. GPT achieves the highest Jigsaw scores in both image and text, while Bagel performs lowest in both. GPT and nano-banana also lead Maze and Sliding path-planning on the strength of their generation. nano-banana's lower Jigsaw score is attributed to its automatic generation producing an incorrect number of images.

• Understand-only models remain competitive or better on several subtasks. Gemini-2.5 Pro shows the strongest reasoning, topping Geometry and the Sliding Puzzle text scores (44.7 step-level / 39.3 sample-level). Unified models generally score lower on Jigsaw than specialized understanding models, largely because intermediate generative steps lengthen their context and degrade format adherence. GPT4.1 reaches 78.0 on Jigsaw text and 56.1/9.4 on Maze.

• Non-understanding generators fail at understanding-driven generation. On the Science task, FLUX.1-Kontext and Imagen 3-001 show the lowest image accuracy, and Imagen 3-001 also records 0.1 for both Code Shape&Color and Position. GPT reaches the highest generation accuracy on Science (61.8 image, 91.1 text result), attributed to world knowledge.

• High-level understanding does not guarantee graphical output ability. Bagel, despite strong understanding, gets low image accuracy on Code Rendering (2.2 Shape&Color, 1.8 Position), while Imagen 4 scores 54.8 (Shape&Color) and 51.6 (Position) — which the authors read as evidence of strong underlying understanding. Imagen 3-001 scores only 0.2 and 0.3.

• Spatial precision is a universal weakness. Across models, positional accuracy on Code Rendering is consistently lower than shape accuracy. In Science, GPT4.1 + GPT-image leads shape (83.6) but position trails far behind (68.6).

• Failures concentrate in intermediate steps. Documented failure modes include instruction-following lapses (adding or omitting elements, rendering text-only fields onto images), background and style drift across edits, fragile spatial perception and world-knowledge application, and topology/semantics violations. Concretely: on Maze, GPT sometimes distorts wall–path topology while Bagel injects extraneous glyphs that make states unparsable; on Jigsaw, OmniGen2 copies the 2×2 reference instead of completing it and nano-banana introduces irrelevant content; on Code, Ovis-U1 and OmniGen2 misread SVG colors, side counts, sizes, or relative positions, and Qwen-Image-Edit rasterizes the text-only "Render Summary" onto the image.

• The evaluator is reliable. Sampled across 150 model outputs from Math, Science, and Code, Cohen's Kappa against Gemini-2.5-pro was 0.7512 for images and 0.7538 for text; against human annotators it was 0.717 for images and 0.7404 for text. On the full 6-level Code Rendering scale (707 samples), Pearson correlation was 0.84 for Shape&Color and 0.76 for Position.

• The benchmark fills a documented gap. Table 1 compares Uni-MMMU against MMMU, WISE, RISEBench, OpenING, MME-Unify, and UniEval across multimodal understanding, generation and editing, multi-turn evaluation, and dual evaluation of process and result — Uni-MMMU is the only one marked as satisfying all four.

Methodology in Plain English

The authors started from the observation that humans solve hard problems by alternating between drawing and thinking, and asked what tasks would force a model to do the same. They selected eight tasks from disciplines where the answer is unambiguous:

• Generation aids Understanding (scaffold tasks). In Maze Navigation, the model gets a 6×6 "perfect maze" and must output alternating move directions and updated maze images until it reaches the goal, ending with the full move sequence as text. Mazes are procedurally generated with Depth-First Search carving and Breadth-First Search verification, keeping only those with shortest paths between 2 and 10 steps. In Sliding Puzzle, a 3×3 8-puzzle in a fixed 9-color palette requires the shortest solution as alternating text moves and images, with BFS used to discard instances that have more than one shortest path. In Geometry, 140 problems from Geo-Laux require drawing auxiliary lines onto a figure and then solving the problem using that self-generated image; a second stage forces the textual solution to be conditioned on the model's own drawing. In Jigsaw, a 2×2 panel with the bottom-right patch removed must be completed with each of two candidate patches from a 3×3 crop of a ShareGPT-4o-Image sample, and the model then reasons over its own two outputs to pick the right one.

• Understanding aids Generation (reasoning-first tasks). Physics, Chemistry, and Biology items are built from LLM-driven generation of initial and final state prompts with manual curation, chosen for deterministic visual outcomes such as thermal expansion, color change, or a plant bending toward light. Code Rendering supplies raw SVG snippets at three difficulty tiers — simple (single primitive), medium (multiple non-overlapping primitives), and complex (overlapping geometry, control flow, symbol reuse) — and asks for both a natural-language description and a faithful rendering.

• Scoring. Intermediate images are parsed by deterministic color-discretization parsers (a 75% majority threshold for maze cells, 80% tolerance for puzzle tiles), yielding sample-level binary scores and step-level fractions. Jigsaw image quality uses the DreamSim metric. Geometry and Science are scored by a Qwen2.5-VL-72B judge, textual solutions by a Qwen3-32B judge, and Code Rendering by a rubric covering semantic correctness of the description plus Shape&Color and Position on a 0–5 scale. Open-weights judges were chosen deliberately for reproducibility, and text and image channels are scored separately so that good rendering cannot mask bad reasoning or vice versa.

• Validation. One hundred fifty outputs across Math, Science, and Code were independently scored by human annotators and by Gemini-2.5-pro using the same rubrics, and agreement was measured with Cohen's Kappa.

Why This Matters

Impact on research. The paper reframes unified-model evaluation from "can it do both?" to "does each ability actually help the other?" Its ablation design — comparing an understanding-only model, a unified model generating its own intermediates, and an oracle with ground-truth intermediates — gives a reusable method for attributing performance to intermediate quality rather than to model scale or world knowledge. The reported finding that generation is the weaker half of current unified models is a concrete target for the field, as is the observation that longer interleaved contexts hurt instruction-following.

Real-world applications implied by the task design:

• Educational technology and tutoring systems, where a model must work through geometry or physics problems step by step and show its work visually rather than only stating an answer.
• Diagram and schematic generation, where a model reads a specification or a short program and produces an accurate drawing — the Code Rendering scenario.
• Spatial planning and navigation assistance, where an agent must maintain and update a map of its environment across many steps.
• Image editing and design tools, where the edit must follow instructions exactly without the background or style drifting from the original.

Industry relevance. The reported failure modes — instruction-following lapses, style drift, spatial imprecision, and errors that corrupt downstream steps — map directly onto the reliability problems that block deployment of generation-plus-understanding systems in production editing, design, and agentic workflows. The authors argue that addressing them will require tighter controllability such as program- or constraint-guided generation, stronger spatial and state invariants across edits, and interfaces that treat executable intermediate representations as first-class parts of the reasoning loop.

Future Directions

• Extending beyond deterministic puzzles and science. The authors state that Uni-MMMU focuses on reasoning-centric disciplines with verifiable solutions and does not cover open-ended creativity, subjective judgment, or nuanced commonsense reasoning.
• Moving from static images to video and longer temporal interactions, which the current benchmark does not cover at all.
• Broadening data curation. Procedural generation guarantees unique solutions and clean parsing but may produce data lacking the complexity, noise, and visual diversity of real imagery; the LLM-driven science pipeline with manual curation could carry subtle biases from generation models or annotators.
• Strengthening the judge pipeline. The model-as-a-judge framework, despite showing substantial agreement with humans, is not infallible and may carry its own biases or knowledge gaps that affect evaluation accuracy — and the paper notes open questions about how to build tighter controllability and stronger state invariants into unified models themselves.

Target Audience

Researchers and engineers working on unified multimodal models, image generation, or vision-language systems who need a benchmark that measures integration rather than isolated capabilities. It is also useful for evaluation and benchmarking specialists interested in dual-level scoring of intermediate and final outputs, for educators or EdTech developers who care about models showing multi-step visual work, and for readers tracking the gap between model understanding and model generation. Readers without a background in multimodal evaluation will still follow the task descriptions, but the results tables assume some familiarity with accuracy-style metrics and model-as-a-judge protocols.

Authors’ abstract

Unified multimodal models aim to jointly enable visual understanding and generation, yet current benchmarks rarely examine their true integration. Existing evaluations either treat the two abilities in isolation or overlook tasks that inherently couple them. To address this gap, we present Uni-MMMU, a comprehensive and discipline-aware benchmark that systematically unfolds the bidirectional synergy between generation and understanding across eight reasoning-centric domains, including science, coding, mathematics, and puzzles. Each task is bidirectionally coupled, demanding models to (i) leverage conceptual understanding to guide precise visual synthesis, or (ii) utilize generation as a cognitive scaffold for analytical reasoning. Uni-MMMU incorporates verifiable intermediate reasoning steps, unique ground truths, and a reproducible scoring protocol for both textual and visual outputs. Through extensive evaluation of state-of-the-art unified, generation-only, and understanding-only models, we reveal substantial performance disparities and cross-modal dependencies, offering new insights into when and how these abilities reinforce one another, and establishing a reliable foundation for advancing unified models.