AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition

AI summary

Overview

Research area: Computer Vision / Vision-Language Models — specifically efficient multimodal inference via adaptive visual token acquisition.

Technical level: Advanced. The paper assumes familiarity with vision-language model architecture, reinforcement learning for LLMs, and GRPO-style policy optimization.

One-sentence scope: AdaptVision is a reinforcement-learning-trained VLM paradigm that lets a model decide per-sample whether to answer from a low-resolution image or spend extra visual tokens cropping a high-resolution region, achieving near-full-resolution accuracy at roughly a third of the visual token cost.

What This Paper Is About

Vision-Language Models burn enormous compute on visual tokens — a single 2048×1024 image produces over 2,600 tokens in Qwen2.5-VL, far exceeding the text tokens in a typical prompt. Existing efficiency methods compress tokens at a fixed ratio, so they treat every question identically: an easy "what color is the sky" query gets the same budget as a dense chart requiring OCR. AdaptVision reframes this as a decision problem, asking whether a VLM can autonomously determine the minimum number of visual tokens each sample needs, using a coarse-to-fine strategy inspired by human active vision.

Key Contributions

1. AdaptVision framework — A VLM paradigm that processes a 1/4-resolution image first (25% of original visual tokens), then either answers directly or invokes a bounding box tool to crop a key region from the high-resolution image for a second reasoning turn.

2. Decoupled Turn Policy Optimization (DTPO) — A new RL algorithm that splits the learning objective into tool learning and answer accuracy, normalizes each token group separately, and computes distinct advantage estimates for tool tokens versus answer tokens.

3. A tailored reward function — Combining an outcome reward (LLM-judged accuracy, format compliance, and a balance penalty) with a tool reward that rewards informative crops while penalizing oversized bounding boxes, discouraging both over-reliance on and avoidance of the tool.

4. Empirical validation — Demonstrating that DTPO-trained AdaptVision achieves 97.9% of vanilla-model average performance using 33% of visual tokens, beating static compression methods and the prior dynamic method VisionThink.

Main Findings

• Substantial token savings at near-full accuracy: AdaptVision retains 33% of the vanilla model's visual tokens while averaging 97.9% of vanilla performance across nine benchmarks (ChartQA, OCRBench, DocVQA, MME, MMVet, RealWorldQA, POPE, MathVista, MathVerse).

• Coarse-to-fine works better than naive downsampling: The down-sample baseline (1/4 resolution, no tool) uses 25% of tokens but only reaches 92.1% relative performance. AdaptVision gains 5.8 percentage points of accuracy for just 7% more tokens.

• Faster inference: End-to-end inference time drops by roughly 1.67× overall versus the vanilla model, despite the extra generated tokens spent on reasoning and tool calls.

• Both reward terms are necessary: Removing the balance reward causes the model to collapse into calling the tool on every sample; removing the tool reward causes it to collapse into direct answering within about 10 training steps.

• DTPO stabilizes training where GRPO fails: Vanilla GRPO initially fails to optimize either reward, then abruptly collapses into excessive tool use around step 20 and calls the tool on 100% of samples. DTPO's rewards rise steadily and converge to a sensible tool-call ratio.

• Adaptive behavior emerges across task types: The trained model calls the tool frequently on fine-grained tasks (MathVerse, ChartQA) and rarely on general ones (POPE), matching the intuition that tools are only useful when detail matters.

• Robust and portable design choices: Performance is stable across hyperparameter settings (λ = 0.2–0.4, α = 1–2), and replacing GPT-4o with a smaller open-source judge (Qwen3-VL-4B) still outperforms VisionThink.

Methodology in Plain English

The researchers start with an off-the-shelf VLM (Qwen2.5-VL-7B-Instruct) and give it a new interaction pattern: every query begins with a quarter-resolution image, and the model can either produce an answer immediately or emit a bounding box marking a region of the original high-resolution image worth inspecting more closely. That crop is fed back as a second turn before the final answer.

To teach the model when to use the tool, they use reinforcement learning. A group of candidate responses is sampled per prompt, and each receives a score. That score mixes an outcome component — whether the answer was judged correct, whether the required <think>/<answer>/<tool_call> formatting was obeyed, and a small penalty for over-using the tool or for "lucky guessing" from low resolution — with a tool component that rewards crops containing the needed information and subtracts a penalty proportional to how large the crop was.

The core algorithmic contribution addresses a flaw in applying standard GRPO to this two-turn setup. GRPO assigns one reward to the entire sequence and normalizes over all tokens equally, which means tool-decision tokens (a small fraction of the sequence) receive weak gradient signal relative to answer tokens. DTPO instead normalizes tool tokens and answer tokens in separate denominators and computes two separate advantages — one from the outcome reward, one from the tool reward — applying the tool advantage only to the tool tokens. This gives each part of the response a signal proportional to its actual role.

Why This Matters

Impact on research: This work is one of the first to apply the "thinking with images" / active perception paradigm — previously used to boost accuracy on hard visual tasks — to the goal of computational efficiency. It shows that tool invocation and token reduction are complementary rather than competing objectives, and DTPO's decoupled advantage scheme generalizes to multi-turn RL scenarios beyond vision.

Real-world applications:

• Document and chart analysis at scale — Processing scanned invoices, financial statements, or scientific figures where only a small region carries the answer, reducing per-page inference cost.
• Mobile and edge deployment — On-device VLMs with tight memory budgets where a fixed high-resolution image would exceed the token limit.
• High-throughput visual QA — Customer support or content moderation pipelines that must handle millions of images per day; a 1.67× speedup translates directly into serving cost savings.
• OCR-heavy workflows — The model only pays for high-resolution detail on the specific documents that need it, rather than uniformly.

Industry relevance: Visual token cost is a dominant driver of VLM serving expense. A method that cuts tokens to a third while preserving accuracy directly reduces GPU-hours, latency, and memory footprint — all first-order concerns for anyone deploying multimodal models in production.

Future Directions

• Broader tool sets and dynamic starting resolution — The current framework uses one tool and a fixed 1/4 initial compression. Letting the model choose its own starting resolution or use multiple tools (e.g., OCR, zoom, region captioning) could improve adaptability further.

• Extending beyond two turns — The maximum of two interaction turns may be insufficient for deeply nested visual reasoning tasks; multi-round tool calling remains open.

• Architecture generalization — The authors state the method is backbone-agnostic but only test it on Qwen2.5-VL. Validating on InternVL, Qwen3-VL, or other families would establish broader applicability.

• Reward model dependence — Crop correctness is currently judged by GPT-4o (though a smaller model works as a substitute). Eliminating the external judge in favor of a learned or self-supervised reward would simplify training.

Target Audience

Researchers and engineers working on efficient multimodal inference, VLM serving infrastructure, and reinforcement learning for language and vision models. It is most valuable to readers already comfortable with GRPO-style policy optimization and VLM architecture, but the framework-level ideas — adaptive token budgets and decoupled multi-turn credit assignment — are accessible to anyone designing production VLM systems.

Authors’ abstract

Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.