OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

AI summary

Overview

Research area: Multimodal large language models (MLLMs), visual tool-use agents, and agentic memory systems (Natural Language Processing / computer vision).

Technical level: Intermediate — readers should know what an LLM orchestrator, an API-style tool call, and a benchmark score are, but no background in training or model internals is required.

Scope: The paper introduces OpenVLHarness, a training-free harness that gives multimodal models a standardized tool layer plus a persistent memory of images, structured data and web references, and reports results across 22 benchmarks and eight backbone models.

What This Paper Is About

Many visual tasks cannot be solved by reasoning harder over the input image alone — they need grounded perception (finding many small objects), external knowledge (looking up a rare entity), or numerical measurement from intermediate values. The authors first show that simply raising a model's inference budget gives only modest returns, then build a harness that instead supplies the missing evidence: a capability layer that wraps heterogeneous tools behind consistent interfaces and renders their outputs into model-readable form, plus a persistent multimodal memory that keeps the underlying artifacts available for later steps. The goal is a general, reusable system that improves visual reasoning without fine-tuning the underlying model.

Key Contributions

1. A capability layer that pairs task-level tool interfaces with model-facing evidence rendering, so heterogeneous backends for perception, search and computation can be swapped without changing the orchestrator's task vocabulary, and their outputs (masks, depth maps, webpages) are converted into interpretable observations.
2. Persistent multimodal memory that registers image artifacts, structured data bindings and web references under reusable handles within a query, letting later tool calls and a Python Coding Agent operate directly on earlier evidence.
3. A large-scale evaluation across 22 benchmarks in four domains (grounded perception, visual search and deep research, general VQA and hallucination, spatial reasoning) with eight open and proprietary backbones, plus component ablations and a failure analysis.
4. Two coding-agent-driven specialization workflows — harness search (editing prompts, tool descriptions and output templates) and tool generation (composing existing capabilities into new tools) — that adapt the system without updating MLLM weights, alongside an open-source release of the harness, tools, evaluation protocols and workflows.

Main Findings

• Reasoning alone plateaus: In a controlled study on 22 benchmarks, moving the base model from xhigh to max reasoning effort multiplies model-token spending by 2.7 but improves the average score by only 0.7 points.
• Broad gains over no-tool baselines: OpenVLHarness improves average scores by 9.7–12.6 points over matched no-tool baselines across eight backbones, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra.
• Specific backbone scores (Table 1): Qwen3-VL-8B rises from 34.24 to 45.41; Qwen3-VL-32B from 38.30 to 50.92; GPT-6 Luna at medium effort from 45.76 to 56.52; GPT-6 Astra at medium effort from 60.27 to 70.17, the highest of any backbone.
• Beats training-free harnesses: The strongest training-free baseline on Qwen3-VL-8B, OctoTools, averages 31.80, and Visual Sketchpad averages 30.49, versus 45.41 for OpenVLHarness — a 13.6-point margin over OctoTools. A coding-agent harness is not automatically helpful: Codex on GPT-6 Luna averages 43.8, below the 45.8 no-tool baseline.
• Beats training-based harnesses without fine-tuning: On Qwen2.5-VL-7B, OpenVLHarness averages 33.45 across the 22 datasets, versus AdaReasoner at 25.29, PixelReasoner at 28.02 and DeepEyesV2 at 25.02, all of which require fine-tuning the backbone.
• More effort does not catch the harness: On the full evaluation splits (26,229 questions per setting), base GPT-6 Luna rises from 44.3 at low effort to 48.0 at max effort, Visual Sketchpad from 44.7 to 48.8, and Codex from 36.9 to 48.6 — all below OpenVLHarness at medium effort (56.5), which incurs at most 37% of their model-token cost. OpenVLHarness at low effort (49.9) is already on par with their max-effort averages.
• Harness and effort are complementary: Raising OpenVLHarness's own effort lifts its average from 56.5 to 60.1; at max effort it leads the best competing method by 15.6 and 12.4 points on grounded perception and visual search/deep research, and by 6.1 and 7.1 on general VQA and spatial reasoning. At low effort it trails the base model slightly on general VQA (61.5 vs. 61.6) and spatial reasoning (54.5 vs. 56.5).
• Both mechanisms matter (ablation): Raw tool access adds +3.8 (8B) and +4.3 (32B) points; the capability layer adds another +3.4 and +3.2; textual memory adds +0.7 and +0.3; completing multimodal memory with visual attachments and reusable image handles adds +3.1 and +4.5, bringing the full harness to 47.14 and 52.51. The final step's largest gains are in visual search and deep research (+6.2 / +3.5) and spatial reasoning (+2.8 / +3.1).
• Failures are domain-specific: A Kimi K3 critic analyzed 9,893 failed Qwen3-VL-8B trajectories, agreeing with human annotations on 92% of 100 sampled failures. In grounded perception, incorrect tool output accounts for 75% of failures, driven by crowded images with more than 25 objects; in visual search and deep research, omitted tool calls account for 49%. Unusable or ignored correct tool outputs are uncommon (3–8% across domains).
• Specialization follows the diagnosis: Harness search improves ERQA by +4.1 and RealWorldQA by +3.7 points where text-level fixes dominate (71% and 56% of proposed remedies), while tool generation yields only +0.2 and +0.1 there. Conversely, tool generation improves SparBench by +8.2 and FREAK-VQA by +4.5 where task-specific tools dominate (75% and 62%), while harness search yields −0.5 and −1.0.

Methodology in Plain English

The system puts an MLLM in the role of an orchestrator that, at each step, either calls a tool or answers. Two mechanisms support it.

The capability layer defines tools by the operation they perform rather than by the model behind them — for instance, Visual Grounding takes an image reference and a text query, and can be backed by SAM 3 or Grounding DINO, while OCR can be backed by PaddleOCR-VL or DeepSeek-OCR. The base instantiation has nine capabilities: five perception tools (Zoom-in, Visual Grounding, OCR, Depth Estimation, Camera Trajectory), three information-seeking tools (Text Search, Image Search, Webpage Visit), and a Python Coding Agent. Each tool also has an output processor that turns raw results into something the model can inspect — a labeled mask overlay with bounding boxes, a colorized depth map, a rendered camera trajectory — while the raw arrays and masks are pushed into memory.

Persistent multimodal memory is an artifact registry per query that separates a payload from its handle. Images get names like input_image_3 and tool_generated_image_6, structured data bindings hold boxes, scores, masks, depth arrays, camera parameters and OCR results, and web references like webpage_url_1 let the model revisit a page without retyping the URL. After each tool turn, an Environment Update advertises what is available. When a later call needs an earlier artifact — say, measuring average depth inside each detected cabinet — the orchestrator hands a natural-language objective to the Python Coding Agent, which writes a program that runs with the stored bindings as named variables.

For evaluation, the authors cover 22 datasets in four domains, compare against no-tool baselines, two training-free harnesses (Visual Sketchpad, OctoTools) and three training-based ones (AdaReasoner, PixelReasoner, DeepEyesV2), and average Qwen3-VL results over three seeds while running proprietary backbones once due to API cost. They then run an effort sweep on GPT-6 Luna from low to max, ablate each mechanism, mine failure trajectories with a critic model, and test two ways of specializing the harness on four datasets with contrasting remedy profiles.

Why This Matters

Impact on research. The paper reframes visual agentic reasoning as an evidence-acquisition problem rather than a pure inference-scaling problem, and provides a controlled demonstration that extra reasoning tokens buy little beyond a point. It also contributes reusable infrastructure: a typed multimodal memory spanning images, structured data and web references, an ablation that isolates the value of rendered evidence versus retained artifacts, and a failure taxonomy that links diagnosis to concrete adaptation strategies.

Real-world applications:

• Document, chart and form understanding where measurements must be computed from OCR and layout data rather than read off a description.
• Deep-research assistants that must find a long-tail entity in an image, search for it, and verify the answer against a retrieved webpage.
• Spatial and embodied tasks requiring metric 3D relations from camera poses, depth maps and grounding masks.
• Domain customization in regulated or specialized settings, where harness search and tool generation adapt behavior without retraining model weights.

Industry relevance. The cost analysis is the practical hook: OpenVLHarness at medium reasoning effort outperforms max-effort prompting and competing harnesses by 7.7–8.5 points while spending at most 37% of their model-token cost, with per-1k-question costs of $0.92 at medium effort versus $0.58 for the bare base model and $1.29–$1.89 for Codex and Visual Sketchpad. That trade-off, plus backend-swappable interfaces, makes the design plausible for production systems that need to change perception vendors without rewriting orchestration logic.

Future Directions

• Crowded-scene perception. Grounded perception remains the weakest domain, with missed or merged instances in images containing more than 25 objects driving 75% of failures; better detection or de-duplication strategies are an open problem.
• Automating specialization. The four-dataset study shows harness search and tool generation help in different regimes, but the critique covers only failed trajectories and offers diagnostic guidance rather than a predicted recoverable accuracy. Turning remedy profiles into an automatic selection procedure is unresolved.
• Reducing token overhead. The harness makes several model calls per question carrying tool outputs and memory; closing the cost gap with single-call prompting while keeping the accuracy gain is a natural next step, and the low-effort results already suggest headroom.
• Extending the evidence surface. The base instantiation uses nine capabilities, and the paper evaluates 6 of a suite of 11 multimodal information-seeking tasks in the main table (the remaining five appear in Appendix B.5), leaving broader tool libraries and additional modalities as open ground.

Target Audience

Researchers and engineers building multimodal agents and tool-use systems, especially those working on inference-time scaling, agentic memory, or benchmark-driven evaluation of MLLMs. It is also useful for practitioners who want a training-free way to add perception, search and computation to an existing model, and for readers interested in how failure analysis can guide targeted system adaptation.

Authors’ abstract

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.