V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval

AI summary

Overview

Research area: Multimodal information retrieval, Multimodal Large Language Models (MLLMs), and agentic tool-use reasoning (Computer Vision / cs.CV).

Technical level: Advanced. The paper assumes familiarity with MLLM architectures, Chain-of-Thought reasoning, embedding-based retrieval, and reinforcement learning (specifically Group Relative Policy Optimization).

Scope: The paper proposes V-Retrver, an evidence-driven retrieval framework that reframes universal multimodal retrieval as an agentic reasoning process in which an MLLM actively calls visual tools to verify fine-grained visual evidence during reranking.

What This Paper Is About

Existing Chain-of-Thought retrieval systems are largely language-driven: they compress visual inputs into fixed embeddings or text descriptions and then reason over them, which produces speculative or hallucinated reasoning when the decisive evidence is visual. V-Retrver addresses this by letting the model alternate between generating a hypothesis and actively inspecting candidate images with visual tools, so that ranking decisions are grounded in verified visual observations rather than static encodings.

Key Contributions

1. An evidence-driven agentic retrieval framework. V-Retrver enables an MLLM to actively acquire visual evidence during multimodal reasoning by invoking external visual tools, reformulating retrieval as an iterative decision-making procedure (hypothesis generation, selective visual inspection, decision refinement) rather than single-pass similarity matching.
2. Multimodal Interleaved Evidence Reasoning (MIER) with two visual tools. The framework interleaves textual reasoning steps with tool invocations and returned visual evidence, using SELECT-IMAGE (choosing a subset of similar candidates for closer inspection) and ZOOM-IN (localized zoom on regions of an image) to perform selective perception.
3. A curriculum-based training strategy with an evidence-aligned RL objective. Three stages — cold-start supervised fine-tuning, Rejection Sampling Fine-Tuning (RSFT), and Evidence-Aligned Policy Optimization (EAPO) instantiated via GRPO — jointly optimize reasoning structure, ranking accuracy, and economical tool use.
4. Broad empirical validation. Experiments on the M-BEIR benchmark and five unseen datasets, plus held-out task and efficiency analyses, show consistent gains in retrieval accuracy, perception-grounded reasoning reliability, and generalization.

Main Findings

• State-of-the-art average recall on M-BEIR. V-Retrver-7B reaches 69.7% average Recall, surpassing U-MARVEL-7B by +4.9% (64.8%) and Retrv-R1-7B by +0.5% (69.2%). The abstract reports 23.0% improvements on average across benchmarks.
• Large gains on fine-grained visual tasks. V-Retrver achieves 51.2% on FIQ and 73.5% on CIRR, versus U-MARVEL-7B at 38.2% and 63.2%. The paper notes Retrv-R1-7B still leads on certain text-centric subtasks (for example F200K and InfoS text-candidate settings), while V-Retrver leads on visually ambiguous tasks requiring fine-grained inspection.
• Strong generalization to unseen datasets. On CIRCO, V-Retrver reaches MAP@5 of 48.2 (vs. MM-Embed 35.5 and LamRA 42.8); on GeneCIS it reaches R@1 of 30.7 (vs. 24.8 for LamRA). It also reports 75.1 R@1 on Visual Storytelling, 31.2 R@1 on Visual Dialog, and 68.3 R@5 on Multi-round FashionIQ.
• Robustness on held-out tasks. Without exposure to three modality-combination types during training, V-Retrver* achieves 61.1% average Recall, outperforming LamRA-7B* (50.9%) by +10.2%, indicating MIER decouples evidence-gathering reasoning from specific input modalities.
• Visual tools matter more than reasoning alone. A text-only RL variant trained identically but without visual tools reaches 61.8% versus V-Retrver's 67.2%.
• Every training stage contributes. Zero-shot tool invocation without alignment collapses to 45.8%, below the 47.2% Qwen2.5-VL-7B baseline. SFT alone reaches 59.4%, SFT+RL without RSFT gives 66.3%, SFT+RSFT without RL yields 60.9%, and the full three-stage pipeline reaches 67.2%.
• All three reward terms are needed. Removing the format reward drops average recall by 1.7% (to 65.5); replacing the contrastive ranking reward with a binary correctness signal causes the largest drop of 2.4% (to 64.8); removing the tool-use reward reduces performance by 1.1% (to 66.1).
• Tool tolerance of 1 is the best trade-off. Fixed τ=1 gives 67.2% average with 1.68 average calls; τ=0 gives 66.8% with 1.12 calls; τ=2 gives 67.5% with 2.45 calls; a dynamic τ gives 67.3% with 2.15 calls. More lenient or dynamic thresholds buy at most 0.3% accuracy while increasing calls by 28% to 46%.
• Learned tool economy. Under EAPO, valid and total tool calls converge during training, response length stabilizes, and the model settles at 1.6 to 1.7 average tool calls per query.
• Competitive efficiency. Against the pointwise baseline LamRA-Rank-Pointwise, which invokes the LLM once per candidate and has an Inference Time Ratio (ITR) of 4.41x at K=50, V-Retrver is substantially faster. Retrv-R1 has a lower ITR (0.22x at K=50) through visual token compression but lower CIRR R@5 (72.3 vs. 73.5, a +1.2% gap for V-Retrver).

Methodology in Plain English

V-Retrver uses a coarse-to-fine pipeline. First, an embedding model encodes the query and all candidates and retrieves the top-K candidates, using the same approach as LamRA for building the embedding model. This shrinks the search space cheaply.

Second, a reasoning agent reranks that shortlist. Unlike a conventional reranker over fixed features, the agent is allowed to call tools. It reasons in text, decides to inspect images, receives the visual evidence back, and revises its ranking. Two tools are available: SELECT-IMAGE picks a subset of candidates when several look semantically alike, and ZOOM-IN magnifies specific regions so the model can examine objects, textures, or spatial layout. This produces a multimodal interleaved reasoning trajectory of text steps, tool requests, visual observations, and a final ranked list.

To train the agent, the authors use a three-stage curriculum. Stage I is a cold-start supervised fine-tuning step using multimodal Chain-of-Thought data synthesized with Qwen2.5-VL-72B-Instruct, filtered by rules to remove malformed or inconsistent samples, which teaches basic reasoning syntax and tool formatting. Stage II is Rejection Sampling Fine-Tuning: many trajectories are sampled per instance and only those that satisfy formatting constraints and produce correct rankings are kept. Stage III is Evidence-Aligned Policy Optimization, an RL objective built on GRPO that scores whole trajectories with a composite reward: a format compliance reward for correct <think>/<answer> tags and valid integer ranking lists, a Contrastive NDCG ranking reward that gives graded credit to placing the correct candidate first and to hard negatives (top-m non-correct candidates by embedding similarity) only when they rank below the correct one, and a tool-use reward that gives credit for successful evidence-based verification while penalizing invocations beyond a tolerance threshold.

The model is initialized from Qwen2.5-VL-7B-Instruct, with the vision encoder frozen across all training stages and the language model fine-tuned. Training uses M-BEIR, comprising eight retrieval tasks across 10 datasets with 1.1M training samples. Evaluation uses Recall@K plus MAP@5 on CIRCO, with K=50 for M-BEIR and K=10 for the unseen datasets. Baselines span foundational VLMs (CLIP, SigLIP, BLIP, BLIP2), fine-tuned universal retrievers (UniIR, MM-Embed, LamRA, U-MARVEL), reasoning-enhanced models (Vision-R1, VLM-R1, Retrv-R1), and agentic tool-use models (DeepEyes).

Why This Matters

Impact on research. The paper argues that retrieval decisions often hinge on visual details that fixed embeddings discard, and that giving a model the ability to "look again" closes a gap left by language-only CoT retrieval. It also shows that agentic tool use, previously explored mainly in other vision tasks, transfers to retrieval, and that alignment via RL is required for tool use to help rather than hurt — zero-shot tool invocation without alignment fell below the untrained baseline.

Real-world applications (as discussed in the paper):

• Retrieval-Augmented Generation over Knowledge-based Visual Question Answering (KVQA) benchmarks, where the paper reports superior retrieval precision and VQA accuracy when retrieval and VQA are trained simultaneously.
• Multimodal recommendation, listed as an intended extension.
• Precision-critical visual domains such as medical imaging and trademark detection, where fine-grained visual discrimination outweighs modest latency overhead.
• Compositional and dialog-style search, covering image-plus-text queries, dialog queries, and multi-interleaved image-text queries evaluated on CIRCO, GeneCIS, Visual Dialog, and Multi-round FashionIQ.

Industry relevance. Systems that rely on MLLM-based retrieval and RAG pipelines can use the approach to improve ranking of visually similar candidates without invoking the LLM once per candidate, as pointwise rerankers do. The release of a project page and Hugging Face model page suggests intended practical reuse.

Future Directions

• Fine-grained credit assignment. The limitations section notes that EAPO evaluates trajectories at the sequence level, making it hard to attribute the contribution of each individual tool call, especially in long reasoning chains; the authors propose exploring fine-grained token-level advantage estimation and step-level credit assignment.
• Lightweight adaptive inference. Multi-turn agentic reasoning with interleaved tool calls and uncompressed visual representations incurs non-negligible latency and memory overhead, motivating early-exit strategies and dynamic tool scheduling for latency-sensitive, large-candidate-pool scenarios.
• Broader downstream tasks. Extending V-Retrver to multimodal recommendation and retrieval-augmented generation is explicitly listed.
• Tool tolerance and verification policy. The threshold ablations suggest open questions remain about how to balance tool-call frequency against accuracy as candidate pools and query types vary.

Target Audience

Researchers and engineers working on multimodal retrieval, MLLM-based reranking, agentic tool-use reasoning, and reinforcement learning for language-and-vision models. It is also relevant to practitioners building retrieval-augmented generation or visual search systems who need to understand the accuracy-versus-efficiency trade-offs of listwise agentic reranking versus pointwise or token-compressed alternatives. Readers without a background in RL-based policy optimization or MLLM training pipelines will find the method sections demanding.

Authors’ abstract

Multimodal Large Language Models (MLLMs) have recently been applied to universal multimodal retrieval, where Chain-of-Thought (CoT) reasoning improves candidate reranking. However, existing approaches remain largely language-driven, relying on static visual encodings and lacking the ability to actively verify fine-grained visual evidence, which often leads to speculative reasoning in visually ambiguous cases. We propose V-Retrver, an evidence-driven retrieval framework that reformulates multimodal retrieval as an agentic reasoning process grounded in visual inspection. V-Retrver enables an MLLM to selectively acquire visual evidence during reasoning via external visual tools, performing a multimodal interleaved reasoning process that alternates between hypothesis generation and targeted visual verification.To train such an evidence-gathering retrieval agent, we adopt a curriculum-based learning strategy combining supervised reasoning activation, rejection-based refinement, and reinforcement learning with an evidence-aligned objective. Experiments across multiple multimodal retrieval benchmarks demonstrate consistent improvements in retrieval accuracy (with 23.0% improvements on average), perception-driven reasoning reliability, and generalization.