MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning

AI summary

Overview

Research area: Multimodal large language model (MLLM) evaluation and reasoning benchmarks, in the Natural Language Processing category.

Technical level: Intermediate. Readers benefit from familiarity with multimodal LLMs, chain-of-thought prompting, and reinforcement-learning-based reasoning methods such as GRPO.

Scope: The paper introduces MMR-Life, a 2,646-question multiple-choice benchmark built on 19,108 real-life images that tests seven types of multimodal multi-image reasoning across 21 tasks, and reports an evaluation of 37 state-of-the-art models plus analyses of thinking length, reasoning-enhancement methods, and cross-type correlations.

What This Paper Is About

Existing multimodal reasoning benchmarks tend to test either expert-level domain knowledge (such as science or mathematics questions) or synthetic symbolic puzzles, and most handle only a single image per question. The authors argue neither matches how people actually reason in daily life, where problems unfold across a sequence of ordinary photographs and require no specialist knowledge. MMR-Life is their response: a benchmark that pieces together multiple real-world images per question and asks models to apply abductive, analogical, causal, deductive, inductive, spatial, and temporal reasoning without relying on domain expertise.

Key Contributions

1. A new benchmark, MMR-Life, described as the first comprehensive benchmark for multimodal multi-image reasoning in real-life scenarios across seven reasoning types, comprising 2,646 multiple-choice questions over 19,108 images, 21 tasks, and 15 image types, with an average of 7.22 images per question.
2. An extensive evaluation of 37 state-of-the-art MLLMs, showing that even the strongest model (GPT-5) reaches only 58.69% accuracy, a 14% gap below human performance (72.28%), with severe bottlenecks in causal, spatial, and temporal reasoning.
3. A paradigm analysis of MLLM reasoning, covering how thinking length, reasoning-enhancement methods (CoT, Self-Consistency, Best-of-N, GRPO), and reasoning type affect performance — including the finding that longer thinking helps only certain reasoning types and that RL generalizes worse than Best-of-N on small models.
4. An error analysis of the two strongest models (GPT-5 and Gemini-2.5-Pro), breaking failures into reasoning, abstraction, knowledge, and perception errors, plus a correlation and hierarchical-clustering study showing that reasoning types form distinct pattern clusters.

Main Findings

• The benchmark is genuinely hard for current models. GPT-5 achieves 58.69% and Gemini-2.5-Pro 56.86%, while humans average 72.28% — a 14% gap. Almost all open-source models score below 40%, and some recent models, such as Skywork-R1V-38B and InternVL3.5-8B, perform worse than random guessing (20%).

• Performance is highly uneven across reasoning types. Models do relatively well on analogical, deductive, and inductive reasoning — most closed-source models outperform humans on analogical reasoning — but struggle badly with causal, spatial, and temporal reasoning. Spatial reasoning is the weakest: the best model reaches only 25.10%, against 79.76% for humans.

• Thinking mode helps closed-source models but not open-source ones. Closed-source thinking models generally beat closed-source non-thinking models, but open-source non-thinking models average 29.01% versus 27.15% for open-source thinking models.

• Reasoning performance scales logarithmically with thinking length, with exceptions. Across the 14 models plotted, longer outputs correlate with higher accuracy, but open-source thinking models such as MiMo-VL-7B-RL and QVQ-72B-Preview fall in the lower-right region of the plot, indicating they spend many tokens without gaining accuracy.

• Longer thinking is not universally better. Comparing CoT versus no-CoT in non-thinking models and varying reasoning budgets (minimal, medium, high) in thinking models shows that inductive reasoning does not improve — and can worsen — with more reasoning, while analogical reasoning clearly benefits.

• Reasoning-enhancement methods lose their advantage as models scale. On Qwen2.5-VL-7B, BoN@8 gives the largest gain over CoT (29.54 vs. 24.68, +4.86). On Qwen2.5-VL-32B gains shrink (BoN@8 30.97, +2.36; GRPO 30.29, +1.68). On Qwen2.5-VL-72B both BoN@8 (39.80, -0.41) and GRPO (39.68, -0.53) fall below plain CoT (40.21).

• RL generalizes worse than Best-of-N on small models. Across all three Qwen2.5-VL scales GRPO trails BoN, and additional experiments on other small MLLMs show RL-trained models consistently underperform BoN inference on their base models — sometimes even underperforming the base model with CoT.

• Reasoning types correlate in structured ways. Pearson correlations range from 0.97 (inductive and analogical) down to 0.40 (spatial and inductive). Hierarchical clustering of these correlations reveals clusters such as Ana–Ind, while spatial reasoning sits far from all other categories.

• Errors are dominated by reasoning failures. In a sample of 140 errors per model (20 per reasoning type) for GPT-5 and Gemini-2.5-Pro, reasoning errors account for 32%, including causal inversion (24%), temporal confusion (42%), and missing key steps (24%). Abstraction errors account for 17%, knowledge errors 17%, and perception errors 12%.

Methodology in Plain English

The authors built the benchmark in four stages. First, they collected real-life images from public image datasets such as Kaggle, open web resources such as the eBird bird-distribution site, public video sources (frames were extracted and checked for clarity), and existing multi-image or video reasoning benchmarks. Second, they designed 21 tasks grouped under seven reasoning types, deliberately excluding purely symbolic images such as charts and puzzles and avoiding tasks that require expert knowledge. Third, they generated question-answer pairs either automatically through heuristic rules (when the images themselves carry the needed information) or through manual annotation (when the reasoning depends on implicit information in the images). Every question is multiple-choice with five options, where each option is either text or an image; negative image options were sampled by heuristic rules, and negative text options were generated by GPT-5-mini, GPT-4o, and Qwen2.5-VL-32B, after which annotators manually picked the four highest-quality wrong answers. Fourth, they filtered data in three ways: difficulty filtering (removing questions that Qwen2.5-VL-7B, Gemma3-4B, and InternVL3.5-8B all answered correctly), format filtering (revising options with large length or format differences to prevent shortcut guessing), and quality filtering (removing ambiguous questions or ones needing domain expertise).

For evaluation, all 37 models received the same zero-shot CoT prompt. Open-source models were run five times and averaged. Experiments used 8 NVIDIA A100 GPUs. Human performance was measured with 12 students of varying degrees and backgrounds, who answered 50 sampled questions at a time from a 210-question mini test set, yielding 600 valid human answers without access to the internet or books.

Why This Matters

Impact on research. The paper argues that existing benchmarks deviate from real-life reasoning: knowledge-heavy benchmarks test expertise that daily life rarely demands, while symbolic-puzzle benchmarks use images that look nothing like real photographs, and most benchmarks use only a single image. MMR-Life reframes evaluation around commonsense-solvable, multi-image, real-world problems, and it provides diagnostic analyses — thinking-length scaling, method comparisons, cross-type correlations — that go beyond a leaderboard score. Because every question is newly annotated rather than sampled from existing datasets, textbooks, or the web, the authors also argue the benchmark reduces data-contamination risk.

Real-world applications:

• Evaluating and selecting multimodal assistants for everyday consumer tasks such as understanding photo albums, recipes, and travel sequences.
• Surveillance and security analysis, exemplified in the paper by tracking recurring individuals across multiple surveillance images to determine event ordering.
• Robotics and embodied AI, where systems must reason about object locations, movement, and spatial relations across a stream of views.
• Video and media understanding pipelines that need to infer causes, effects, and event order from image sequences.

Industry relevance. The findings give model developers concrete signals: spatial and temporal reasoning remain largely unsolved (best model 25.10% on spatial versus 79.76% human), long chain-of-thought is not a universal fix, and reinforcement learning on small models may not generalize better than cheaper inference-time methods like Best-of-N. The paper also notes that open-source thinking models average below open-source non-thinking models on this benchmark, pointing to a gap in generalization to real-world contexts.

Future Directions

• Improving abstract world representations. The authors state that models struggle with spatial and temporal reasoning and that this bias is one future model training should correct, since current models acquire analogy and deduction more easily.
• Rethinking reinforcement learning on small models. The paper explicitly leaves open the question of whether RL methods on small models merely overfit to specific datasets, given that GRPO trails BoN across scales and architectures.
• Balancing reasoning efficiency and effectiveness. The lower-right placement of models such as MiMo-VL-7B-RL and QVQ-72B-Preview shows that long outputs do not guarantee accuracy, a problem the authors frame as a major challenge for future open-source MLLMs.
• Exploiting the hierarchy of reasoning patterns. The clustering results suggest a higher-level structure of reasoning patterns; the authors present this as a foundation for building more generalizable AI systems and for understanding how reasoning transfers across tasks.

Target Audience

Researchers and engineers working on multimodal large language models, benchmark design, or evaluation methodology; practitioners choosing or fine-tuning MLLMs for real-world image-sequence tasks; and readers interested in how chain-of-thought length, reinforcement learning, and inference-time search affect reasoning generalization. The paper is most useful to those already comfortable with MLLM terminology, though its benchmark framing and results are accessible to a broader technical audience.

Authors’ abstract

Recent progress in the reasoning capabilities of multimodal large language models (MLLMs) has empowered them to address more complex tasks such as scientific analysis and mathematical reasoning. Despite their promise, MLLMs' reasoning abilities across different scenarios in real life remain largely unexplored and lack standardized benchmarks for evaluation. To address this gap, we introduce MMR-Life, a comprehensive benchmark designed to evaluate the diverse multimodal multi-image reasoning capabilities of MLLMs across real-life scenarios. MMR-Life consists of 2,646 multiple-choice questions based on 19,108 images primarily sourced from real-world contexts, comprehensively covering seven reasoning types: abductive, analogical, causal, deductive, inductive, spatial, and temporal. Unlike existing reasoning benchmarks, MMR-Life does not rely on domain-specific expertise but instead requires models to integrate information across multiple images and apply diverse reasoning abilities. The evaluation of 37 advanced models highlights the substantial challenge posed by MMR-Life. Even top models like GPT-5 achieve only 58% accuracy and display considerable variance in performance across reasoning types. Moreover, we analyze the reasoning paradigms of existing MLLMs, exploring how factors such as thinking length, reasoning method, and reasoning type affect their performance. In summary, MMR-Life establishes a comprehensive foundation for evaluating, analyzing, and improving the next generation of multimodal reasoning systems.