VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents

AI summary

Overview

• Research area: Multimodal agents, open-web information seeking, and benchmark design for vision-language models (Computer Vision / agentic search).
• Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with tool-using agents (ReAct-style loops), multimodal LLMs, and benchmark evaluation conventions such as Pass@1 and partial-credit scoring.
• Scope in one sentence: The paper introduces VisSearch-Bench, a 204-task human-written benchmark that tests whether multimodal agents can organize and reuse visual information across long, multi-step open-web searches, and reports results for 16 multimodal models with and without search tools.

What This Paper Is About

Most existing multimodal search benchmarks only check whether an image was used somewhere in a task, and in many of them the image mainly serves to name a starting entity after which plain text search suffices. The authors argue this misses the harder skill of "visual context management": tracking multiple images, linking visual details to external facts, and letting earlier visual findings determine later search targets. VisSearch-Bench is built to measure that skill directly, across five task families and 7 main domains (25 subdomains), with tasks that require combining object appearance, local details, or spatial relations with information found on the web.

Key Contributions

1. A task-based framing of visual context management. The paper defines visual context as the visual information in a task plus its links to object identities, external facts, and search targets, and evaluates how agents organize, connect, and reuse it across search steps rather than merely whether images appear in a prompt.
2. VisSearch-Bench itself: 204 human-written tasks across 7 primary domains and 25 secondary domains, organized into five categories — Set-wise Visual Search (68 tasks, 33.3%), Sequential Visual Search (34), Hybrid Wide–Deep Search (34), Competitive Visual Disambiguation (34), and Visual-Structure-Guided Search (34, each 16.7%).
3. A human-led construction pipeline with automated verification. Annotators fix public sources first, build traceable evidence structures, and write questions that hide intermediate nodes and the final answer; automated checks test source accessibility, field completeness, set coverage, answer recomputation, and possible shortcuts, followed by human review of every retained task.
4. A two-metric evaluation protocol. Full-answer Accuracy is paired with a task-specific Process Score that measures the fraction of labeled critical states (individual results and subquestion answers) recovered, so partial progress and its location can be distinguished from complete failure.

Main Findings

• Even the best system is far from solving the benchmark. Across 16 multimodal models, the highest accuracy is 26.47% (ReAct with Kimi-K3), meaning the best system fully solves only about a quarter of the 204 tasks even with search tools.
• Direct answering collapses. Without tools, no model exceeds 6.37% accuracy. Accuracy and Process Score are highest for GPT-5.5 and Gemini 3.5 Flash among the closed-source direct-answer settings (both 6.37% accuracy; Process Scores of 22.61% and 22.28% respectively).
• Accuracy and Process Score diverge. ReAct with GPT-5.5 reaches 25.49% accuracy but the highest Process Score at 49.52%, while ReAct with Kimi-K3 leads on accuracy. Solving whole tasks does not track performance on individual results and subquestions.
• Search helps, but the framework interacts with the backbone. Qwen3.5-397B-A17B goes from 1.96% accuracy without tools to 15.69% with ReAct and 10.29% with DeerFlow, and its Process Score rises from 12.72% to 32.43% and 28.37%. For Qwen3.6-35B-A3B the ordering reverses: DeerFlow gives 10.78% accuracy and 24.26% Process Score versus 7.35% and 23.25% for ReAct.
• Set-wise search shows the widest gap between partial and complete success. The best accuracy in this category is 11.76% (ReAct with Kimi-K3), yet ReAct with GPT-5.5 reaches a 44.10% Process Score there with only 8.82% accuracy.
• Competitive Visual Disambiguation defeats direct answering entirely. All direct-answer models score 0.00% accuracy; ReAct with DeepSeek-V4.1-Flash and ReAct with Qwen3.8-27B both reach 44.12%.
• Model strengths vary by category. ReAct with Kimi-K3 leads Sequential Visual Search (32.35% accuracy, 56.34% Process Score); ReAct with GPT-5.5 leads Hybrid Wide–Deep Search (23.53% accuracy, 48.36% Process Score) and Visual-Structure-Guided Search (50.00% accuracy, 70.61% Process Score).
• Direct image access beats caption-mediated access, but captions rescue some tasks. Switching from Full to Caption loses 8, 10, and 20 correct answers for Qwen3.6-35B-A3B, Qwen3.5-397B-A17B, and DeepSeek-V4.1-Flash respectively, while gaining 6, 8, and 13 tasks those models fail under Full. Overall Caption loses more than it gains: Qwen3.5-397B-A17B drops from 15.69% to 14.71% accuracy and from 32.43% to 25.42% Process Score.
• Visual tool use is front-loaded but not confined to the start. Across models, 41.5%–69.2% of all visual tool calls occur in the first 20% of the search process, yet these tools continue to be used in later stages.
• More tool calls do not mean higher accuracy. The paper reports that tool choices, search rounds, and runtime differ across models, and that greater call volume does not reliably correspond to better performance.

Methodology in Plain English

Each task gives an agent a natural-language question, an optional set of starting images (present in 187 of 204 tasks, i.e. 91.7%; absent in 17), and a required answer format. The agent can call five tools — web search, webpage reading, text-based image search, reverse image search, and image cropping — and must produce an answer in that format.

To build tasks, annotators started from public sources that offer a bounded set of entities, dependable entity–image links, and independently checkable facts. They fixed those sources before writing questions, so reference answers do not depend on what a model happens to retrieve. They then arranged entities, images, facts, and candidate sets into a traceable evidence structure and imposed the structural constraints of one of the five task families, writing a question that hides intermediate nodes and the final answer. Automated checks caught broken sources, incomplete fields, missing set members, unreproducible answers, and shortcuts that would let a model skip the intended visual work; humans then reviewed every surviving task. Each item ships with evidence sources, a locked reference answer, and scoring rules, and the dataset totals 3,593 reference-solution steps (median 17.0 and mean 17.6 per task).

Scoring uses two numbers. Accuracy is strict: the predicted answer must contain all and only the required content, with alias normalization, order ignored for unordered sets, and order preserved for sequences. Process Score breaks each task into m_i labeled critical states and gives credit for the fraction recovered, matched by field label with aliases treated as equal — so different search paths score the same if they recover the same states. Direct-answer models are scored identically, which makes the with-tools versus without-tools comparison fair.

Inference was constrained and uniform: tools return at most five results per search, webpage text is truncated to 12,000 characters, images are capped at 1,536 pixels on the longest side, there is a budget of 20 images per task shared across initial images, retrieved images, and crops, crops can be enlarged up to four times, and each task allows 30 ReAct rounds and 60 evidence-tool calls at no more than four per round. If a limit is hit, one extra model call submits the final answer from whatever evidence exists. Every system is run once per task from a fresh context and reported as Pass@1. Forward web and image search use Serper; reverse-image search uses SerpApi.

Why This Matters

Impact on research. The benchmark shifts attention from "was an image involved?" to "was visual information carried forward correctly?" That reframing matters because a single aggregate accuracy number hides whether a failure came from searching the wrong space, missing a visual candidate, or mis-aggregating evidence — which is exactly what the Process Score is designed to expose. The large accuracy–Process Score gaps (for example 8.82% versus 44.10% on Set-wise Visual Search) give the field a concrete diagnostic rather than a leaderboard scalar.

Potential real-world applications.

• Shopping and catalog verification: confirming that every item in a set meets a visual condition, then attaching correct facts to each item — the scenario the paper uses to motivate Set-wise Visual Search.
• Product and collectible authentication: distinguishing near-identical objects by local shape, texture, markings, or individual parts, which is the Competitive Visual Disambiguation setting.
• Research and archival work: following visual clues from an artwork or artifact to records about its creator, collection, or exhibitions across chained searches.
• Technical inspection and field work: using object order, spatial position, or part connections in an image to decide what to look up next, rather than treating each image as an isolated recognition problem.

Industry relevance. The work comes from Tsinghua University, Ant Group, and Renmin University of China, and the tool stack (web search, image search, reverse image search, cropping) mirrors what commercial browsing agents actually have. The finding that no single agent framework wins for every backbone is directly actionable for teams choosing between ReAct-style and DeerFlow-style orchestration, and the front-loaded visual tool usage statistic is useful for anyone budgeting image calls in a production pipeline.

Future Directions

• Improving set-level completion. The persistent gap between partial credit and full success on Set-wise Visual Search suggests that verifying many objects and keeping individual judgments consistent with a final summary is an unsolved sub-problem, not just a scaling issue.
• Better use of visual evidence in later search stages. Since a substantial share of visual tool calls already comes late in the process, the open question is how to convert that late-arriving evidence into correct, complete answers rather than merely gathering it.
• Designing frameworks that adapt to the backbone. No single framework was best across all models tested with both ReAct and DeerFlow, which invites work on framework selection or on frameworks that adapt their tool policy to the underlying model.
• Understanding when captioning helps. Caption-mediated access recovered 6, 8, and 13 tasks that direct image inspection failed, yet lost more than it gained overall; characterizing which task types benefit from an intermediate description step is an open question. The paper does not report human performance baselines, inter-annotator agreement statistics, or error taxonomies, all of which would sharpen future comparisons.

Target Audience

Researchers and engineers working on multimodal agents, tool-using LLM systems, and open-web search or browsing agents will get the most from this paper, particularly those building evaluation suites or choosing between agent frameworks. It is also useful for vision-language model developers who want to see where current models break on multi-image and multi-object tasks, and for practitioners in e-commerce, catalog QA, or digital archiving who need agents to verify sets of visually similar objects rather than answer single-image questions. Readers need only moderate background: comfort with benchmark metrics and ReAct-style tool loops is sufficient.

Authors’ abstract

Multimodal agents use web and image tools to answer questions beyond their internal knowledge. Some search tasks use images mainly to identify an initial entity, then rely on text retrieval. These tasks offer limited insight into how agents connect information across images, objects, and search steps. We therefore focus on visual context management. This involves organizing and using visual information and linking it to other findings during search. Visually intensive tasks require agents to combine object appearance, local details, or spatial relations with external information. We introduce VisSearch-Bench, with 204 human-written tasks across 7 main domains and 25 subdomains. It covers Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation, and Visual-Structure-Guided Search. Each task provides a reference answer, traceable evidence, and scoring rules. Accuracy measures complete task success, while Process Score measures the correctness and completeness of individual results and subquestion answers. Across 16 multimodal models, the best system fully solves only 26.47% of tasks, even with search tools. These results highlight the challenge of linking visual evidence with search results to produce complete answers.