MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation

AI summary

Overview

Research area: Evaluation of multimodal generative models — specifically, whether text-to-image and text-to-video systems perform genuine reasoning or merely produce visually plausible media. Sits at the intersection of NLP, computer vision, and AI benchmarking.

Technical level: Intermediate. Readers need basic familiarity with diffusion/video generators, vision-language models, and benchmark design, but no deep mathematical background is required.

Scope: The paper introduces MMGR, a cross-modal benchmark of 1,853 task instances across 10 tasks, 3 domains, and 5 reasoning abilities, evaluated with answer-verifiable and process-aware scoring across video generators, image generators, and LLM/VLM baselines.

What This Paper Is About

Modern image and video generators produce convincing visuals from text prompts, but visual fluency does not prove that a model can reason. A generated video can look smooth while breaking causal order, object permanence, or task rules, and a generated image can look plausible while violating a logical or spatial constraint. MMGR asks whether multimodal generators can actually reason through the images and videos they produce, and builds a benchmark that scores outputs on whether they solve the underlying reasoning problem rather than on how realistic they look.

Key Contributions

1. A cross-modal benchmark spanning three domains. MMGR unifies Abstract Reasoning (Maze, Sudoku, ARC, Math), Embodied Navigation (Last-Mile, Top-down, 3D Real-World, SLAG), and Physical Commonsense (Physical Concepts, Sports) into one framework, letting video, image, and LLM/VLM generators be compared on identical task families — something previous modality-specific benchmarks could not do.

2. A process-aware chain-of-frame criterion for video. For video generation, intermediate frames must form valid steps toward the target outcome, not merely smooth transitions between plausible states. This distinguishes genuine step-by-step reasoning from decorative interpolation.

3. Fine-grained, gated scoring instead of holistic judgment. Each task decomposes into metrics with three roles — integrity/constraint preservation, outcome correctness, and reasoning-process validity — and the primary score is 1 only when all required criteria hold. A single constraint violation yields 0.

4. Hybrid deterministic and VLM-based verification with a reported reliability analysis. Exact rule-based checkers (pixel-based for Maze, OCR-based for Sudoku) are used wherever ground truth permits; a Gemini-2.5-Pro judge with task rubrics handles the rest, and human-annotator agreement is reported per task to bound confidence in the results.

Main Findings

• A sharp gap separates visual quality from reasoning correctness. Video models are strongest where tasks resemble natural video dynamics but collapse on symbolic tasks: all video models score 0.00% on Sudoku, under 12% on ARC, and under 11% on Math.

• Sora-2 leads Physical Commonsense. It reaches 76.00% on Physical Concepts and 64.00% on Sports, with Veo-3 following at 41.67% and 60.00%. Because this domain relies on VLM judging rather than a deterministic checker, these scores reflect judged physical plausibility rather than verified physical correctness.

• Image generators beat video generators on embodied navigation. Nano-banana Pro leads all four navigation tasks — Last-Mile (75.83%), Top-down View (33.05%), 3D Real-World (85.00%), and SLAG (37.29%) — despite producing no temporal output. Longer, more visually coherent generation does not automatically yield stronger reasoning.

• Video models degrade as viewpoint and horizon change. Veo-3 reaches 60.00% on short-horizon Last-Mile navigation but falls to 19.49% on Top-down View and 11.02% on SLAG, exposing failures in maintaining geometry, agent identity, and goal state across frames.

• Math is the notable exception for image models. Nano-banana Pro reaches 72.69%, close to text-only Gemini baselines (71.38% and 74.14%), while video models remain at 0.00–10.61%.

• Evaluator reliability varies widely by task. Pixel-based Maze agreement with human labels is 95.56–100%, OCR-based Sudoku is 100%, and ARC is 94.90%. VLM-based judging is far less stable for embodied navigation (21.67–88.45%) and only moderate for physics (65.31%), so exact model rankings in those domains should be read cautiously.

• Failures share one diagnosis. Generators are better at local visual plausibility than at maintaining a persistent task state. Mazes cross walls or stop short; Sudoku outputs alter clues or break row/column rules; ARC outputs corrupt the demonstration context; navigation videos hallucinate extra agents, shift geometry, or drift from the goal.

Methodology in Plain English

The authors built a standardized test suite of 1,853 task instances and ran each one through state-of-the-art generators in a zero-shot setting — five outputs per prompt, no task-specific tuning — using default settings for closed models and recommended settings for open ones.

Task design. Each task has a controllable difficulty ladder. Mazes vary generator algorithm (DFS and Wilson), grid size (3×3 to 13×13), and start-goal placement. Sudoku varies grid size (4×4 and 9×9) and clue sparsity. Navigation tasks vary floor complexity, view fidelity, path length, and goal specification across 24 hard-level configurations.

Scoring. Rather than asking a judge for one holistic verdict, the authors break each task into specific criteria and score each separately. The final score is gated: all criteria must pass for a sample to count as a success.

Verification. Where exact ground truth exists, they use deterministic checkers — pixel-level path reconstruction for Maze, OCR plus constraint checking for Sudoku — applied frame by frame for videos and to the single output for images. Where no exact verifier exists (navigation, physics), they use Gemini-2.5-Pro as an automated judge with task-specific rubrics.

Reliability check. Three graduate-student annotators independently scored balanced subsets of outputs at $25/hour, with at least two annotators per item and disagreements resolved by discussion. Agreement between human and automatic labels is reported per task to calibrate how much confidence each conclusion deserves.

Cost. The benchmark cost roughly $46k in inference, dominated by video generation (Veo-3 about $30k, Sora-2 about $9k), plus about 1,440 A100 80GB GPU-hours for the two open-source models.

Why This Matters

Impact on research. Prior generative-model benchmarks were fragmented: video suites emphasized quality, alignment, or physical plausibility; image suites tested commonsense or editing; and almost all collapsed evaluation into a single final output judged holistically. MMGR shifts the question from "does this look real?" to "does this solve the problem?" and its cross-modal design lets researchers separate reasoning failures from modality-specific rendering failures — a distinction a video-only benchmark cannot make. Its inclusion of real-world embodied navigation, which concurrent process-aware benchmarks like VIPER and VBVR do not cover, broadens the scope further.

Real-world applications:

• Robotics and embodied agents. Navigation tasks probe whether generated trajectories stay consistent with an environment, goal, and viewpoint — directly relevant to planners that must produce valid action sequences, not just plausible-looking paths.
• Education and tutoring systems. Math and ARC tasks test whether a model's step-by-step solution is correct at each intermediate stage, which is what matters when a system explains reasoning to a student rather than just stating an answer.
• Simulation and synthetic data generation. Physical Commonsense results indicate which generators can be trusted to produce physically plausible footage, informing whether generated video is safe to use for training or scenario simulation.
• Model selection and procurement. The benchmark gives practitioners task-specific, difficulty-controlled evidence about which generator to deploy for a given reasoning-heavy workload, rather than relying on visual-quality leaderboards.

Industry relevance. The paper reports that video models score 0.00% on Sudoku and that image models outperform video models on navigation, which is directly actionable for teams deciding whether to invest in video generation pipelines for tasks that are fundamentally about constraint satisfaction and spatial consistency. The reported cost breakdown (roughly $46k and 1,440 GPU-hours) also gives a concrete sense of the compute economics of rigorous generative-model evaluation.

Future Directions

1. Generators that explicitly optimize for task state. The authors argue progress requires training signals and architectures that reward state preservation, constraint satisfaction, cross-view consistency, and completed trajectories — not just realistic appearance — alongside longer and more controllable temporal horizons.

2. Training data beyond salient physical-commonsense clips. Current video training data skews toward everyday physical scenes; the paper calls for data covering embodied, navigational, and interaction-heavy scenarios to close the reasoning gap.

3. Evaluation that can verify task state. Benchmarks should pair deterministic and process-level metrics with trajectory-consistency tests and failure-specific measures, reducing reliance on VLM judges whose agreement with humans drops as low as 21.67% on some navigation subsets.

4. Extending beyond zero-shot discrete reasoning. The limitations section flags untested territory: fine-tuned and task-specific models, non-text conditioning such as video-to-video editing, continuous control, long-horizon planning, and open-ended creative generation, plus real-world settings with ambiguous goals or partial observability.

Target Audience

Researchers and engineers working on multimodal generative models, benchmark design, or evaluation methodology will get the most from this paper, particularly those who need to distinguish reasoning capability from visual fidelity. It is also useful for applied practitioners selecting image or video generators for reasoning-heavy tasks, and for anyone studying the reliability limits of VLM-as-judge evaluation, since the paper quantifies where automatic scoring can and cannot be trusted. Readers without a background in generative models will find the task descriptions and failure case analyses accessible, though the scoring details and appendix material assume some familiarity with benchmark construction.

Authors’ abstract

Modern multimodal generative models can synthesize visually compelling images and videos, but it remains unclear whether this visual fluency reflects genuine reasoning: when prompted to generate a solution, can a model preserve the physical, logical, spatial, and temporal constraints a task requires, or does it merely produce plausible-looking media? To answer this question, we introduce MMGR (Multi-Modal Generative Reasoning Benchmark and Evaluation), a benchmark for evaluating generative reasoning across video, image, and language-based systems. MMGR covers 10 tasks from three domains (Abstract Reasoning, Embodied Navigation, and Physical Commonsense) and probes five reasoning abilities: Physical, Logical, 2D Spatial, 3D Spatial, and Temporal. Its evaluation emphasizes answer-verifiable tasks and, for video generation, process-aware chain-of-frame reasoning, where intermediate frames must form valid steps toward the target outcome rather than visually smooth but incorrect transitions. Evaluating state-of-the-art video generators, image generators, and LLM/VLM baselines reveals a sharp gap between visual quality and reasoning correctness: video models perform best on Physical Commonsense, but remain weak on symbolic tasks such as Sudoku, ARC, and Math, and brittle in cross-view embodied navigation. Image generators often outperform video generators on embodied navigation despite lacking temporal outputs, showing that longer visual generation does not automatically yield stronger reasoning. MMGR reframes evaluation of multimodal generation from whether outputs look realistic to whether they solve the underlying reasoning problem.