MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

AI summary

Overview

• Research area: Multimodal retrieval-augmented agents (deep research systems that interleave visual/texual reasoning with web search), with a focus on reinforcement learning and distillation for agent training.
• Technical level: Advanced — assumes familiarity with GRPO-style policy optimization, on-policy distillation, and supervised fine-tuning of LLM/VLM agents.
• Scope (one sentence): The paper introduces MMSearch-OPD, a training recipe that recycles existing easy multimodal question–image pairs into harder long-horizon search questions (the "Refine" pipeline) and trains a search agent with combined reinforcement learning and selectively applied on-policy distillation from a larger teacher.

What This Paper Is About

Training multimodal search agents for long-horizon investigations — where the agent must chain many searches and read retrieved images to gather evidence — is limited by two things: existing multimodal datasets are mostly solvable with the model's own knowledge or one or two searches, and end-of-trajectory answer rewards give almost no feedback about how the agent searched. The authors solve the data problem by taking existing easy image–question pairs and mechanically stretching their reasoning chains, and they solve the feedback problem by blending answer rewards and efficiency penalties with token-level guidance from a larger teacher model, applied only to the student's own imperfect trajectories and gradually faded out.

Key Contributions

1. The Refine data pipeline. A procedure that reuses existing simple multimodal questions and their candidate reasoning chains, then applies three configurable operators — Expand (append L web-verified hops), Blur (attach W disambiguating facts to M internal nodes), and Visualize (attach up to P retrieved images and Q verified visual facts to N nodes, selected from the terminal node backward) — before synthesizing a new question that reuses the original image.
2. A joint RL + on-policy distillation (OPD) training objective. Answer correctness is combined with a non-positive behavior penalty H_i (penalizing inefficient tool use and repetitive search); GRPO-style group-centered advantages are supplemented with token-level teacher guidance that is weighted by the shaped reward gap ω_i = [1 − R_i]₊ and applied only to rollouts with c_i < 1 and valid teacher scores.
3. An annealing schedule for teacher influence. The distillation coefficient λ_s starts at 0.05 and decays by a factor of 100 every ten RL steps, so imitation pressure from the SFT teacher disappears as outcome-based optimization takes over.
4. A quantified notion of search difficulty and a quality-control stage. Difficulty is expressed as an evidence-burden sum D_Refined over Expand, Blur and Visualize terms; synthesized questions are then validated by hand-editing for solvability, filtered by a no-tool direct-answer probe, and probed with k tool-augmented expert rollouts (Pass@k) to test learnability.

Main Findings

• Large gain over the same base model's agentic workflow. MMSearch-OPD-9B (RL) reaches 60.2 average accuracy on seven benchmarks versus 44.9 for the Qwen3.5-9B agentic workflow, a 15.3-point improvement; the paper reports this as the highest accuracy among compared 7B–9B search agents on all seven benchmarks, exceeding DeepVoyager-VL-8B (50.6) by 9.6 points.
• The trained 9B beats an untuned 27B workflow. The 9B student's 60.2 average is 3.5 points above the Qwen3.5-27B agentic workflow (56.7); the 27B SFT teacher reaches 63.2, a 6.5-point gain over its own workflow baseline.
• Gains concentrate on the harder, longer-horizon suites. Average improvement over the same 9B workflow is 12.8 points on MMSearch, LiveVQA and BrowseComp-VL, and 17.2 points on MM-BrowseComp, MMSearch-Plus, BrowseComp-V3 and VisBrowse-Bench. Per-benchmark 9B RL scores: MMSearch 86.3, LiveVQA 89.0, BC-VL 67.7, MM-BC 42.0, MMSearch+ 47.0, BC-V3 37.3, VisBrowse 52.1.
• Joint training adds value beyond SFT. The 9B SFT checkpoint lifts average accuracy from 44.9 to 57.9; subsequent joint RL–OPD training raises it to 60.2 (+2.3), improving every benchmark.
• Refined data alone improves SFT results. With identical initialization, SFT config and training budget, refined questions beat the original seeds on SimpleVQA (79.0 vs 72.7, +6.3), FVQA (76.0 vs 74.0, +2.0), VDR (35.8 vs 29.0, +6.8) and BrowseComp-V3 (33.0 vs 18.0, +15.0).
• Refined data lengthens searches substantially. Average tool calls per trajectory rise from 6.61 to 33.71 (counting each search query separately); per question, ImageSearch calls go from 0.38 to 4.88 and FetchImage calls from 0.51 to 3.47.
• Each Refine operator makes the task harder in a distinct way. On 100 paired seed questions, expert accuracy falls from 79% to 65% (Expand), 44% (Blur), and 33% (Visualize); average turns rise from 10.2 to 15.1 after Blur and to 20.2 after Visualize, with tool calls reaching 34.7 per question and ImageSearch + FetchImage share of tool actions rising from 6.3% to 47.5%.
• RL improves answers and then shortens searches. Mean answer reward before behavior penalties rises from 0.347 to 0.587 over 19 steps; comparing steps 6–13 with 14–19, tool calls drop from 26.0 to 21.3 (18.3%) and response length from 29.0k to 23.4k tokens (19.5%) while mean answer reward stays comparable (0.525 vs 0.544).

Methodology in Plain English

Step 1 — Start from questions the model already finds easy. The authors take the full training splits of LiveVQA and FVQA and keep only questions that a data-generation model (Seed2.1-Pro) answers correctly without tools. Because the model succeeds, its chain-of-thought is available as a candidate reasoning route.

Step 2 — Turn that route into an explicit entity chain. A component called BuildLink extracts a sequence of entities from the input image to the final answer, recording the linking clue for each transition. This strips away free-form reasoning so the authors can operate on individual hops.

Step 3 — Stretch the chain in three controllable ways. Expand checks each existing hop against web evidence, throws out chains that fail, and appends exactly L new evidence-supported hops — the old answer becomes a waypoint and the new final entity becomes the answer. Blur picks M interior nodes and attaches W distinct, source-grounded facts to each, so the entity can no longer be named directly. Visualize picks N nodes counting back from the end and attaches up to P verified retrieved images with Q visual facts each, so the agent must find and read an image mid-search.

Step 4 — Write the question last. A question is drafted with entity names spelled out, then names at blurred or visualized nodes are replaced with descriptions built from those facts and images. This two-stage synthesis keeps the chain, the answer, and the original image intact.

Step 5 — Check the questions are real. Humans walk each question from the image, fix wording only minimally, and verify chain consistency, evidence provenance, image–text alignment, and actual use of the added evidence. Questions still answerable by a no-tool direct probe are discarded, and an expert model runs k tool-augmented rollouts (Pass@k) to confirm the questions are learnable and to rank difficulty.

Step 6 — Train in two regimes. Curated questions are split into an SFT subset and a harder RL subset. A Qwen3.5-27B model is fine-tuned on successful trajectories from both and frozen as the teacher; Qwen3.5-9B is SFT-initialized on the first subset, then trained with RL + OPD on the second. During RL, each question gets G sampled rollouts; the reward is correctness plus a non-positive efficiency penalty, centered within the group. The teacher additionally scores each token of the student's own trajectory, and that per-token signal is scaled by how far the rollout's shaped reward is from 1 and is switched off entirely for fully correct rollouts. The distillation weight decays exponentially so the student ends up optimized by outcome rewards rather than by imitation.

Evaluation. Seven primary benchmarks (MMSearch, LiveVQA, BrowseComp-VL, MM-BrowseComp, MMSearch-Plus, BrowseComp-V3, VisBrowse-Bench), judged by Qwen3.6-35B-A3B with the Vision-DeepResearch judge prompt, three judgments averaged per prediction, and a cap of 50 interaction turns per question. Tooling: ms-swift for SFT, veRL for joint RL–OPD.

Why This Matters

Impact on research. The paper reframes data scarcity in agentic multimodal search as an upcycling problem: rather than building new multi-hop graphs with relation collection, image curation and evidence validation, it shows that existing image–question pairs can be pushed into long-horizon territory by parameterized transformations, with the difficulty increase measured (accuracy 79% → 33%) rather than assumed. It also offers a concrete recipe for the sparse-reward problem — use the reward gap, not just correctness, to decide how strongly and where to distill a teacher, and anneal that guidance away.

Real-world applications.

• Fact-checking and verification workflows that must locate a specific entity in a retrieved image before the next textual query can even be formed.
• Knowledge-intensive visual customer support and product identification, where an answer depends on cross-referencing a photo against live web evidence.
• Research assistants for scientific, legal or due-diligence investigation that must sustain dozens of lookups without stopping prematurely.
• Multimodal RAG systems for e-commerce and media archives, where the agent has to decide which image to inspect and when to stop verifying.

Industry relevance. The headline operational result is that a 9B model trained this way (60.2 average) exceeds an untuned 27B model's agentic workflow (56.7) — an argument for investing in training data and post-training over raw parameter count. The efficiency dynamics matter too: later RL steps cut tool calls by 18.3% and generated tokens by 19.5% at comparable reward, which maps directly onto inference cost and latency for deployed search agents.

Future Directions

• Shortcut leakage. The authors state that filtering cannot rule out all shortcuts from parametric knowledge or alternative web evidence, so the necessity criterion in the paper holds only for the designated chain; stronger adversarial filtering is an open problem.
• Noisy teachers and judges. Training quality depends on the teacher (Qwen3.5-27B) and the answer judge (Qwen3.6-35B-A3B); quantifying and mitigating that noise is left open.
• Reproducibility of open-web search. Changing web content and tool availability limit reproducibility, which invites cached or snapshot-based evaluation protocols.
• Component attribution in the RL run. The reported reward and length dynamics are described as batch-level trends of the joint run and explicitly do not isolate the effect of individual components, leaving a clean per-component analysis (behavior penalty vs. OPD vs. annealing schedule) for follow-up work.

Target Audience

Researchers and engineers working on agentic multimodal retrieval, deep-research agents, and RL post-training for tool-using models; practitioners who already have a search-agent pipeline and want a data-refinement and training recipe that does not require building a new dataset from scratch; and readers interested in on-policy distillation schedules for long-horizon, sparsely rewarded tasks. Readers without background in GRPO-style optimization or distillation will find the method sections demanding, but the Refine pipeline and the ablation tables are accessible on their own.

Authors’ abstract

Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.