AdaTooler-V: Adaptive Tool-Use for Images and Videos

AI summary

Overview

• Research area: Multimodal machine learning — specifically tool-augmented reasoning in multimodal large language models (MLLMs) covering both images and video.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning for LLMs (GRPO, policy gradients), multimodal chain-of-thought, and vision tool integration. The conceptual motivation, however, is accessible to anyone who has used a vision-language model.
• Scope: Introduces AdaTooler-V, a 7B MLLM trained to decide adaptively when vision tools help versus when plain text reasoning suffices, with a new RL algorithm (AT-GRPO), two training datasets, and evaluations across twelve benchmarks.

What This Paper Is About

Recent multimodal models interleave vision tool calls (image crops, video frame extraction) into their reasoning chains. However, most open-source models invoke these tools indiscriminately, even on questions answerable with plain reasoning, which inflates inference cost and can actually degrade accuracy by inducing "overthinking" and pulling the model away from its optimal reasoning path. This paper's goal is to teach an MLLM to invoke vision tools only when they produce genuine gains, using a reinforcement learning signal tied to the measured benefit of tool use per sample.

Key Contributions

1. AdaTooler-V, a multimodal LLM with adaptive tool-use capability, which switches between pure text-based chain-of-thought and multimodal interleaved reasoning depending on whether a problem actually benefits from vision tools.
2. AT-GRPO (Adaptive Tool-use GRPO), a reinforcement learning algorithm that rescales rewards per sample using a Tool Benefit Score, penalizing unnecessary tool calls while rewarding genuinely helpful ones via a Gaussian decay on tool frequency.
3. Two large-scale training datasets — AdaTooler-V-CoT-100k for SFT cold start and AdaTooler-V-300k for RL — spanning single-image, multi-image, and video data across math, counting, logic, spatial, chart, OCR, and knowledge tasks.
4. Comprehensive empirical validation across twelve benchmarks, including a 7B model reaching 89.8% on V*, surpassing GPT-4o and Gemini 1.5 Pro.

Main Findings

• High-resolution image reasoning: AdaTooler-V-7B hits 89.8% on the V* benchmark, exceeding Pixel Reasoner (84.3%), DeepEyes (85.6%), Mini-o3 (88.2%), and improving over the base Qwen2.5-VL-7B by +11.3 points.
• General image reasoning gains: Improvements on MME (2460.8), InfoVQA (86.0), MMBench (87.8), MathVista (74.5), and multi-image tasks MMSI-Bench (36.8) and SPAR-Bench (40.3).
• Video benchmarks: With 32 frames, AdaTooler-V reaches 46.7% on VSI-Bench, 54.6% on VideoMMMU, and 68.4% on MVBench; performance scales with frame count up to 128 frames. On Video-Holmes, it scores 55.6% versus 27.8% for the base model.
• Value of AT-GRPO: Ablations show SFT+AT-GRPO averages 69.9 across four benchmarks versus 67.6 for SFT+standard GRPO and 65.9 for GRPO alone.
• Necessity of SFT cold start: Skipping SFT degrades performance because the model lacks structured tool-use priors, producing incoherent trajectories in early RL stages.
• Value of tools themselves: Disabling tool use (RL without tools) drops V* from 89.8% to 84.4% and VSI-Bench from 46.7% to 39.9%, confirming vision tools supply complementary evidence to text reasoning.
• Response length decreases during RL: Training curves show response length falls sharply in early RL, evidence that the model learns to skip unnecessary tool calls on simple problems.
• Low sensitivity to reward weight α: α = 0.6 works best, but 0.6 and 0.8 give comparable results, indicating robustness in that range.

Methodology in Plain English

The pipeline begins with a user query plus image or video. The policy model decides: for straightforward questions, it produces a single internal thought and answers directly; for harder questions, it enters a loop of thoughts, tool actions, and observations until it answers or hits a turn/length limit. Four tools are available: CropImg (zoom/crop), FrameAt (single video frame), VideoClip (video segment), and PathTracer (draw trajectories between points).

Training happens in two phases. First, supervised fine-tuning on AdaTooler-V-CoT-100k teaches the model coherent multi-turn tool-interaction trajectories, providing structured priors. Second, reinforcement learning on AdaTooler-V-300k with the new AT-GRPO algorithm refines its strategy.

The key trick is the Tool Benefit Score (ΔS), computed per sample as the accuracy gap when the same question is answered with tools versus without, using eight runs each from Qwen2.5-VL-72B-Instruct. AT-GRPO multiplies this score by a Gaussian factor on tool frequency, so if tools hurt a sample (ΔS < 0), each additional tool call increases the penalty; if tools help (ΔS > 0), calls are positively rewarded but moderated so excessive calls are discouraged. The combined reward — correctness/formatting plus a weighted adaptive-tool term — feeds into the standard GRPO advantage calculation and policy update with a KL penalty against a reference model.

Why This Matters

• Research impact: Challenges the assumption that more tool use is always better in multimodal reasoning, reframing tool invocation as a decision that should be optimized alongside answer accuracy. The Tool Benefit Score concept offers a reusable signal for other agentic vision models.
• Real-world applications:
  • Mobile and edge assistants where every unnecessary tool call costs latency and battery.
  • Document, chart, and form analysis where most queries need only text reasoning.
  • Clinical or industrial video inspection where selective frame extraction matters for throughput.
  • Multi-image reasoning for retail, real estate, or insurance comparison tasks.
• Industry relevance: A 7B open-weight model outperforming proprietary systems like GPT-4o on a high-resolution benchmark lowers the cost barrier for deploying tool-augmented MLLMs. Reducing redundant tool calls directly translates into cheaper inference at scale.

Future Directions

1. Better Tool Benefit estimation: ΔS currently relies on a single reference model, which may be biased; learned estimators or model ensembles could yield more reliable scores.
2. Reward design for open-ended generation: The current verifiable reward scheme favors multiple-choice and numerical tasks; learned reward models or multimodal discriminators could extend adaptive tool use to free-form outputs.
3. Broader data coverage: The dataset is mostly built from public benchmarks, limiting long-tail, noisy, and cross-domain generalization; in-the-wild samples and hard-case synthesis are natural next steps.
4. Scaling and cross-model generalization: Whether AT-GRPO transfers to larger backbones or other tool spaces (retrieval, code, external APIs) remains an open question.

Target Audience

Researchers and engineers working on multimodal LLMs, agentic vision systems, or reinforcement learning for language models. It is also valuable for practitioners building cost-sensitive vision assistants who need to balance tool-driven accuracy against inference overhead, and for readers tracking the emerging "thinking with images" line of work.

Authors’ abstract

Recent advances have shown that multimodal large language models (MLLMs) benefit from multimodal interleaved chain-of-thought (CoT) with vision tool interactions. However, existing open-source models often exhibit blind tool-use reasoning patterns, invoking vision tools even when they are unnecessary, which significantly increases inference overhead and degrades model performance. To this end, we propose AdaTooler-V, an MLLM that performs adaptive tool-use by determining whether a visual problem truly requires tools. First, we introduce AT-GRPO, a reinforcement learning algorithm that adaptively adjusts reward scales based on the Tool Benefit Score of each sample, encouraging the model to invoke tools only when they provide genuine improvements. Moreover, we construct two datasets to support training: AdaTooler-V-CoT-100k for SFT cold start and AdaTooler-V-300k for RL with verifiable rewards across single-image, multi-image, and video data. Experiments across twelve benchmarks demonstrate the strong reasoning capability of AdaTooler-V, outperforming existing methods in diverse visual reasoning tasks. Notably, AdaTooler-V-7B achieves an accuracy of 89.8\% on the high-resolution benchmark V*, surpassing the commercial proprietary model GPT-4o and Gemini 1.5 Pro. All code, models, and data are released.