Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents

AI summary

Overview

Research area: Evaluation of multimodal large language models (MLLMs) as agents acting through external tools, specifically a new capability the authors call dexterous visual tool use — fine-grained, closed-loop parameterized visual action.

Technical level: Advanced. The paper assumes familiarity with multimodal agents, tool-use benchmarks, parameterized action spaces (Bézier stroke rendering), LoRA fine-tuning, and trajectory-level diagnostics such as AUC and similarity curves.

Scope: The paper introduces EASEL, a benchmark of 110 reference-guided reconstruction samples plus 5 instruction-conditioned semantic tasks, together with EASEL-Data (440k supervision samples) and EASEL-9B, and uses them to evaluate 25 multimodal agents on precision-critical visual action.

What This Paper Is About

Existing agent benchmarks — software engineering, web navigation, desktop GUI use, and API calling — are precision-tolerant: an action succeeds as long as it lands inside a generous bounding region, so exact spatial control is never strictly required. Other visual systems sidestep precision differently, either by helping the model understand an image without producing persistent visual output, or by delegating execution to a separate segmentation or generative model. This paper asks what happens when neither escape hatch exists: the agent must read pixel-level visual cues, infer explicit tool parameters, and have those parameters directly determine the visible result, updated and corrected step by step. EASEL operationalizes this through painting — reference-guided visual reconstruction — because each stroke visibly changes the canvas and parameter errors cannot be absorbed by a tolerant interface.

Key Contributions

1. EASEL, a benchmark for dexterous visual tool use. It frames the capability as fine-grained, closed-loop parameterized visual action, and consists of 110 reference-guided reconstruction samples plus 5 instruction-conditioned semantic tasks, covering up to 11,392 interactions per full evaluation.

2. EASEL-Data and EASEL-9B. A scalable two-stage curriculum pipeline converts stroke-based rendering trajectories into 440k next-action supervision samples (C1: early reconstruction, 308k samples; C2: light completion, 132k samples), plus 44k chain-of-thought samples covering 10% of references. EASEL-9B is trained on this data to probe whether trajectory supervision helps.

3. A two-axis evaluation protocol. Every rollout is scored on Result Quality (whether the final output matches the target) and Trajectory Quality (how the model uses intermediate feedback), the latter via Similarity@50%, Trajectory AUC, Best Similarity, and Final-Best Gap.

4. A systematic evaluation of 25 multimodal agents (6 closed-source APIs, 19 open-source models from 4B to 90B parameters) that surfaces two distinct closed-loop failure modes and sharp capability boundaries on semantic tasks that reconstruction metrics alone do not reveal.

Main Findings

• Reconstruction performance is low overall and clearly stratified. Final Similarity bottlenecks in the 0.40–0.54 range across evaluated agents. Gemini 3.1 Pro leads at 0.535, followed by Claude Opus 4.7 at 0.472; open-source models cluster in the 0.40–0.45 range with small inter-model variance and no clear correlation with parameter count.

• The task is not inherently unsolvable at that score range. Score anchors show a blank white canvas at 0.446, a reference mean-color fill at 0.534, and a reference dominant-color fill at 0.531 (all uniform fills score 0.000 Edge IoU). The data policy reaches 0.665 under a budget-aligned setting (Edge IoU 0.281) and 0.715 in its native 250-step setting (Edge IoU 0.563) — both substantially above every evaluated agent.

• Two distinct closed-loop failure modes. Gemini 3.1 Pro, Claude Opus 4.7, and Claude Sonnet 4.6 rise sharply within the first 5% of steps and then plateau, with near-identical @50%, AUC, and Final scores (early saturation). GPT-5.5, GLM-5V-Turbo, and qwen3.5-plus keep acting but degrade, with AUC exceeding Final in all three cases; GPT-5.5 shows the strongest degradation (Final-Best Gap 0.073, AUC 0.440 vs. Final 0.426). Across all closed-source models, the effective peak is reached within the first 10% of the step budget.

• Small Final-Best Gaps can mean inaction, not skill. The five open-weight models with Final-Best Gap below 0.01 have exact-repeat rates of 82.3%–92.4%, so their small gaps reflect near-total inaction rather than active quality maintenance.

• The bottleneck is sustained correction, not initial perception. Closed-source plateau models reach comparable Final-Best Gaps (≤ 0.035) through selectively conservative actions, implying genuine sensitivity to canvas state, and the gap between closed-source plateau models (0.47–0.54) and open-source near-inaction models (0.40–0.45) suggests stronger models read visual evidence more effectively early. Yet even top closed-source models plateau far below the data policy's 0.665–0.715.

• Category difficulty follows a clear gradient. Natural Images is hardest (complex textures and fine-grained color distributions) and Programmatic is easiest (strong geometric priors). Gemini 3.1 Pro scores 0.598 on Programmatic versus 0.472 on Natural Images.

• Semantic tasks expose boundaries reconstruction hides. Gemini 3.1 Pro leads outline tasks (Circle 0.919, Apple 0.783) and achieves perfect handwriting (1.000), but scores only 0.183 on Simple Maze and 0.007 on the harder Maze. GPT-5.5 achieves a perfect 1.000 on Simple Maze despite weaker reconstruction, while the harder Maze remains unsolved across all models (GPT-5.5: 0.031).

• Action strategy stability is a prerequisite. Claude Opus 4.7 fails Circle Outline entirely (0.000) due to persistent draw-undo loops, with 124 non-maze undos versus only 3 maze undos. Claude Sonnet 4.6 almost never uses undo yet produces a 0.93 invalid-action rate on maze tasks — it neither detects nor responds to constraint violations.

• Closed-loop self-correction is possible but rare. GPT-5.5 shows iterative correction on Simple Maze, submitting after 25 undos with a perfect score — the clearest example of closed-loop self-correction in the evaluation. Gemini 3.1 Pro and GPT-5.5 use undo almost exclusively on maze tasks, suggesting targeted collision recovery.

• Trajectory supervision helps modestly, with early training doing most of the work. Qwen3.5-9B (base) scores 0.432; EASEL-9B trained on C1 only reaches 0.456 (+0.024); the full C1+C2 curriculum reaches 0.459 (+0.027 absolute, a relative 6.3%). C2 contributes a further +0.003. Final-Best Gap rises slightly from 0.015 (base) to 0.019–0.020 (trained).

• EASEL-9B ranks third overall and is the highest-scoring non-proprietary model on the reconstruction task.

• Visual perception is preserved after training. On standard perception benchmarks, EASEL-9B matches or modestly improves on its base model: MMVP +0.7%, POPE +2.0%, HallusionBench +0.6%, and BLINK spatial reasoning +0% and counting +1.7%.

Methodology in Plain English

The researchers needed a task where an agent's own parameters are the result, with no tolerant interface or external model absorbing mistakes. Painting fits: the agent sees a reference image and the current canvas, then emits a JSON brush_stroke action defining a quadratic Bézier path plus appearance settings. The renderer composites that stroke, the canvas updates, and the loop repeats.

The stroke action has 13 continuous normalized parameters: six path coordinates (start, control, end), per-endpoint radius and opacity (linearly interpolated along the stroke), and a shared RGB color. The arcade semantic tasks use a reduced schema that fixes appearance and requires only the six path coordinates. Coordinates are quantized to integers in [0, 1000] to align with MLLM tokenization.

The benchmark has 110 reconstruction samples in four categories forming a difficulty gradient — Programmatic (15 samples, 64×64, budget 32), Spatial Geometry (40, 128×128, budget 96), Abstract/Cartoon (15, 128×128, budget 96), and Natural Images (40, 128×128, budget 128) — plus 5 semantic tasks (two region-annotation, one handwriting, two path-planning). Programmatic, maze, and Circle Outline tasks are procedurally synthesized; the remaining categories are generated via GPT-image-2.

Scoring runs on two axes. Result Quality uses Final Similarity S_T, a weighted composite of 0.5 × SSIM + 0.3 × (1 − normalized RGB L1) + 0.2 × Edge IoU. Trajectory Quality adds Similarity@50%, Trajectory AUC, Best Similarity, and Final-Best Gap. Semantic tasks score by Dice coefficient (outlines), multimodal LLM letter recognition (handwriting), and normalized path progress (mazes).

For training data, the authors collected 11k reference images (procedural geometry and text/glyphs, COCO subject crops, Tiny ImageNet images), generated a 250-step stroke trajectory per reference using a stroke-based painting policy, and converted trajectories into next-action predictions by replaying prior actions to produce canvases. The curriculum keeps all early strokes (0–15) and samples through stroke 49 for C1, then spans the full trajectory with decreasing density for C2. EASEL-9B is Qwen3.5-9B fine-tuned in two LoRA stages (rank 64, alpha 128, all-linear targets) at learning rates 1×10⁻⁴ and 5×10⁻⁵ with 15% C1 replay, using bf16, DeepSpeed ZeRO-2, effective batch size 64 on 8 GPUs, max sequence length 8192, and AdamW with weight decay 0.1 and a cosine schedule.

Why This Matters

Impact on research. The paper separates a capability that existing evaluations conflate: precise control of action parameters versus high-level task completion. By showing that the strongest models plateau within the first 10% of the step budget and that semantic performance is partly orthogonal to reconstruction precision (GPT-5.5 scores a perfect Simple Maze while underperforming on reconstruction), it argues that answer-only QA and high-level tool-use benchmarks hide failures that precision-critical settings expose. The finding that trajectory supervision modestly improves fine-grained perception (MMVP +0.7%, POPE +2.0%, HallusionBench +0.6%) hints that EASEL-style tasks could serve as a training signal.

Real-world applications. The authors motivate the broader capability through domains that share the same structure — visual state observation, structured action prediction, and closed-loop feedback:

• Medical annotation, where a clinician-facing agent would mark precise regions itself rather than delegating to a segmentation model.
• CAD sketching, where parameter precision directly determines the geometry produced.
• Diagram authoring, extending the paper's own semantic tasks to structured technical content.
• Embodied manipulation, where the same read-act-observe loop governs physical outcomes.

Industry relevance. EASEL's traces and environment feedback make it compatible with outcome-based optimization, reinforcement learning, and planning methods, and the released artifacts (GitHub repository and Hugging Face benchmark page) give developers a controlled protocol for diagnosing fine-grained visual manipulation in products where approximate answers are not good enough.

Future Directions

• Sustained feedback-driven correction. The authors identify this as the actual bottleneck: models read visual evidence reasonably well early but cannot maintain corrective action across a long trajectory. Closing the gap to the data policy's 0.665–0.715 is the central open problem.

• Better supervision targeting long horizons. C1 training alone accounts for most of EASEL-9B's gain (+0.024 of +0.027), and C2 adds only +0.003 while Final-Best Gap slightly worsens. Designing supervision that improves mid-to-late refinement is explicitly left open.

• More targeted training-signal design. The paper raises the possibility that EASEL-style tasks, combined with better-designed supervision, could train fine-grained visual perception generally — only the first probe of this idea is reported.

• Closed-loop annotation without dedicated segmentation models. The paper claims its annotation setting is, to its knowledge, the first closed-loop evaluation where general-purpose agents perform segmentation-style action without dedicated segmentation models; extending that interface to harder, real annotation domains is a natural next step.

Target Audience

Researchers and engineers building or evaluating multimodal agents — particularly those working on tool use, GUI/web navigation, and visual grounding who want to know where precision-tolerant benchmarks stop being informative. It is also relevant to practitioners in annotation, design tooling, and robotics who need agents that produce exact spatial output rather than approximate placements, and to anyone designing training data or curricula for closed-loop visual action. Readers should be comfortable with MLLM evaluation methodology and parameterized action formats; the paper is not an introduction to multimodal models.

Authors’ abstract

Evaluation is shifting from static QA toward agentic settings where models act through external tools. We identify a critical yet underexplored capability within this space - dexterous visual tool use: fine-grained, closed-loop parameterized visual action in which models infer tool parameters from visual evidence, and those parameters directly govern the final result. Existing benchmarks cover web navigation, GUI operation, and software engineering, but rarely target this coupling between visual evidence and execution precision. We propose EASEL, a benchmark evaluating a controlled instance of dexterous visual tool use that adopts reference-guided visual reconstruction as its primary proxy task: the agent incrementally paints a canvas to match a reference image. EASEL additionally includes semantic tasks spanning region annotation, handwriting, and path planning. We further provide EASEL-Data, a 440k-sample two-stage curriculum dataset for trajectory supervision, and EASEL-9B to investigate its effect on this capability. Evaluation of 25 models reveals that current multimodal agents systematically struggle on EASEL. Reconstruction similarity bottlenecks at low levels (0.40-0.54), while trajectory diagnostics expose severe closed-loop instability - models typically saturate early or degrade post-peak. Semantic tasks reveal sharp capability boundaries in precision annotation and path planning. EASEL-9B, trained on EASEL-Data, surpasses the base model by a relative 6.3%, ranking third among all evaluated models.