MMhops-R1: Multimodal Multi-hop Reasoning

AI summary

Overview

Research area: Multimodal large language models (MLLMs), knowledge-based visual question answering (KB-VQA), retrieval-augmented generation (mRAG), and reinforcement learning for agentic reasoning.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning objective functions (PPO/DAPO/GRPO-style policy optimization), retrieval-augmented generation, and multimodal model training.

Scope: The paper introduces MMhops, a 31,117-sample benchmark for multimodal multi-hop reasoning requiring external Wikipedia knowledge, and MMhops-R1, a reinforcement-learning-trained mRAG framework that lets a model autonomously plan retrieval and reasoning paths across image and text retrievers.

What This Paper Is About

Existing knowledge-based visual QA datasets usually need only one visual recognition step plus one text retrieval step, so today's MLLMs are effectively single-step reasoners that never learn to build long reasoning chains. The authors build a benchmark whose questions force multi-hop chains across images and text, then train a model with reinforcement learning so it can decide on its own when to look at another image, when to query a text retriever, and when to answer.

Key Contributions

1. MMhops benchmark. The authors present MMhops, described as the first large-scale benchmark for multimodal multi-hop reasoning, requiring synthesis of visual and textual information across varying reasoning depths. It has two task formats, Bridging (single image, chain reasoning) and Comparison (multiple images, cross-image comparative reasoning).

2. MMhops-R1 framework. A multimodal RAG framework that uses reinforcement learning to optimize a policy model so it dynamically interacts with multiple retrievers and adaptively plans its reasoning path, rather than following a fixed retrieval pipeline.

3. Composite reward design. The framework uses three reward components: an outcome reward (answer correctness), a format reward (correct tag structure), and an action reward (effective tool use, gated by both correctness and format). It also masks externally retrieved tokens out of the loss computation.

4. State-of-the-art results plus cross-dataset generalization. MMhops-R1 outperforms four categories of baselines on MMhops and generalizes to single-hop INFOSEEK questions and two-hop E-VQA questions.

Main Findings

• MMhops scale and difficulty: The dataset contains 31,117 VQA samples, of which 26,437 (85.0%) are Bridging and 4,680 (15.0%) are Comparison. It involves 20,483 unique questions, 8,832 entities, and 28,256 images. All 31,117 samples (100.0%) require external knowledge; 22,016 samples (70.8%) require 3 steps and 9,101 samples (29.2%) require 4 steps. Average question length is 17.3 words and average answer length is 1.6 words. Answer types are 5,923 entity-type (19.0%), 5,016 temporal (16.1%), and 20,178 numerical (64.9%).

• Comparison against prior datasets: In the paper's comparison table, OK-VQA has 14K samples, A-OKVQA 24.9K, ViQuAE 3.7K, INFOSEEK 1.35M, and E-VQA 1M, but all have total reasoning depth of 2–3 and none require multi-image input. MMhops is listed with 31.1K samples, 1–2 visual reasoning steps, 2–3 text reasoning steps, total reasoning of 3–4 steps, multi-image input, and Wikipedia as the knowledge source.

• Main benchmark results: MMhops-R1 (built on Qwen2.5-VL-7B-Instruct) reaches 51.35% overall on Bridging (String 44.66, Numerical 55.33, Time 47.94) and 22.01% on Comparison. For reference, GPT-4o scores 36.62% overall on Bridging and 8.76% on Comparison; Gemini-2.5-pro scores 53.98% and 29.39%; OmniSearch with GPT-4o scores 42.65% and 17.02%; the 72B Qwen2.5-VL-Instruct baseline scores 34.39% and 7.59%.

• Open models lag despite size: Even the 72B Qwen2.5-VL model falls short of the 7B-based MMhops-R1 by 16.96% and 14.42% in overall accuracy on Bridging and Comparison questions respectively.

• Text-only RAG is insufficient: Self-Ask with GPT-4o, a strong text multi-hop method, actually drops below plain GPT-4o on Comparison questions in MMhops (30.42% vs. 36.62%), which the authors attribute to the lack of visual access.

• Multimodal RAG baselines are also limited: Even OmniSearch with GPT-4o remains 9.7% and 4.99% lower than MMhops-R1 in overall accuracy on Bridging and Comparison questions respectively.

• Closed-source models remain the ceiling but are not solved: Gemini-2.5-Pro outperforms MMhops-R1 but answers only about half of the Bridging questions correctly, with lower accuracy on Comparison questions.

• Cross-dataset generalization (INFOSEEK): MMhops-R1 scores 33.8 on unseen questions, 32.6 on unseen entities, and 33.2 overall, compared to CLIP-PaLM (22.7/18.5/20.4), CLIP-FiD (23.3/19.1/20.9), Wiki-LLaVA (30.1/27.8/28.9), and EchoSight (overall 31.3; unseen-question and unseen-entity scores not reported).

• Cross-dataset generalization (E-VQA): On two-hop questions, MMhops-R1 scores 23.3 versus PaLI 14.7, PaLM 22.8, and GPT-3 18.7.

• Reward ablations matter, especially on Comparison: Removing the action reward drops Bridging overall from 51.35 to 47.57 and Comparison from 22.01 to 20.62. Removing the format reward drops Bridging to 49.97 and Comparison to 14.42. Removing both drops Bridging to 41.75 and Comparison to 13.03.

• Four interaction rounds is the sweet spot: With a maximum of 2, 3, 4, and 5 retriever interactions, Bridging overall scores are 39.93, 47.97, 51.35, and 51.92 respectively, and Comparison scores are 9.83, 13.78, 22.01, and 20.09. The paper reports no significant gains from going to 5 rounds, only extra computation.

Methodology in Plain English

Building the data. The authors start from existing single-hop knowledge-based VQA data and grow it outward. For Bridging questions, they repeatedly take an answer that is a Wikipedia entity, use a language model to write a new question about that entity, verify the new question is unanswerable unless the entity is named (quality control), and then splice the two questions together so the intermediate entity is replaced by a descriptive phrase. Repeating this yields two-hop and three-hop chains. For Comparison questions, they collect visual entities from Wikipedia, use NV-Embed-v2 embeddings to find similar entities, deduplicate with an LLM, match paragraphs containing quantifiable numbers, generate comparative questions, validate them, replace entity names with images, and normalize numeric answers with units stated in the question. GPT-4o was used for the annotation steps.

Training the model. The model is given two tools, an image retriever and a text retriever, and a small action vocabulary: think, image search, text search, answer. At each step it thinks, then chooses an action; retrievals are wrapped in tags such as <image_search> and <text_search>, and the final answer goes in <answer> tags. Malformed actions get a fixed penalty. A trajectory ends at the answer action or after a maximum number of steps.

Reward design. Three signals are combined. Correctness of the final answer gives an outcome reward. Correct tag structure gives a format reward. Tool-use effectiveness gives an action reward, but only if the answer is both correct and correctly formatted — the authors describe this gating as encouraging tool use that actually contributes to success. Weights α=1.0, β=1.0, γ=0.25 combine them.

Optimization. The training objective is adapted from DAPO with dynamic sampling: each group of 8 responses must contain at least one correct sample. Tokens that came from retrievers or environment feedback are masked out of the loss so only the model's own reasoning and action tokens are optimized.

Setup. Training uses the Verl framework with Qwen2.5-VL-7B-Instruct as the backbone, one epoch, constant learning rate 1×10⁻⁶, batch size 256, and group size 8. The retrieval knowledge base is 100K Wikipedia articles each paired with an image. Image retrieval uses CLIP-ViT-L/14@336px; text retrieval uses E5 to fetch the top-3 passages. The maximum number of interaction turns is 4 during both training and inference. Evaluation follows the INFOSEEK protocol: Exact Match for string answers, EM with ±1 year tolerance for time answers, and ±0.1 margin or at least 50% IoU for numerical answers, with an overall weighted average. Data is split 7:1:2 into train/validation/test via stratified sampling on reasoning depth and question type.

Why This Matters

The paper argues that real-world problems rarely resolve in one retrieval step, and that the field's benchmarks have been too shallow to force models to learn genuine multi-step, cross-modal reasoning. By providing a harder benchmark plus a training recipe, it shifts mRAG from fixed pipelines toward learned, dynamic policies.

Potential applications suggested by the paper's framing (the paper does not enumerate specific deployments):

• Knowledge-intensive visual assistants that must look up facts about what a user shows them and chain follow-up questions.
• Comparative analysis across multiple images, such as weighing the quantitative attributes of two entities.
• Open-domain question answering over large corpora where the correct number of retrieval steps varies by question.
• Any setting where a system must decide on its own how many images and documents to consult before answering.

Industry relevance: The main result — a 7B model beating a 72B general-purpose model on this task — suggests that task-specific reinforcement learning over retrieval tools can substitute for raw scale. That is directly relevant to teams building retrieval-backed assistants who care about inference cost and controllable tool use. Three of the author affiliations are industrial (Tencent Inc., PeopleAI Inc.), and the code, dataset, and model weights are stated to be released.

Future Directions

• Closing the gap to closed-source models. Gemini-2.5-Pro still leads MMhops-R1 on both Bridging and Comparison, and even it answers only about half the Bridging questions correctly. What capability is missing is not reported.
• Scaling beyond 4 steps. Five interaction rounds produced no meaningful gain over four here, but whether harder data would reward longer chains is untested.
• Reducing reliance on synthetic annotation. The dataset is built with GPT-4o-generated and quality-filtered questions; the paper does not report human verification rates or error analysis of generated questions.
• Extending the RL paradigm further into multimodal tool use. The authors frame this as the first RL-based multimodal multi-hop framework, leaving open whether richer tool sets, more retrievers, or different backbones improve results.

Target Audience

Researchers and engineers working on multimodal LLMs, retrieval-augmented generation, knowledge-based VQA, and reinforcement learning for tool-using agents. It is also useful for practitioners evaluating whether a small fine-tuned model can match much larger general-purpose models on knowledge-intensive multimodal tasks. Readers without a background in RL policy optimization or RAG pipelines will find the methodology sections dense.

Authors’ abstract

The ability to perform multi-modal multi-hop reasoning by iteratively integrating information across various modalities and external knowledge is critical for addressing complex real-world challenges. However, existing Multi-modal Large Language Models (MLLMs) are predominantly limited to single-step reasoning, as existing benchmarks lack the complexity needed to evaluate and drive multi-hop abilities. To bridge this gap, we introduce MMhops, a novel, large-scale benchmark designed to systematically evaluate and foster multi-modal multi-hop reasoning. MMhops dataset comprises two challenging task formats, Bridging and Comparison, which necessitate that models dynamically construct complex reasoning chains by integrating external knowledge. To tackle the challenges posed by MMhops, we propose MMhops-R1, a novel multi-modal Retrieval-Augmented Generation (mRAG) framework for dynamic reasoning. Our framework utilizes reinforcement learning to optimize the model for autonomously planning reasoning paths, formulating targeted queries, and synthesizing multi-level information. Comprehensive experiments demonstrate that MMhops-R1 significantly outperforms strong baselines on MMhops, highlighting that dynamic planning and multi-modal knowledge integration are crucial for complex reasoning. Moreover, MMhops-R1 demonstrates strong generalization to tasks requiring fixed-hop reasoning, underscoring the robustness of our dynamic planning approach. In conclusion, our work contributes a challenging new benchmark and a powerful baseline model, and we will release the associated code, data, and weights to catalyze future research in this critical area.