Evolve Vision-Language-Action Model into an Agent with On-the-fly Tool-use

AI summary

Overview

• Research area: Robotics, specifically Vision-Language-Action (VLA) models and agentic tool-use for robot manipulation.
• Technical level: Advanced.
• Scope: The paper proposes ART (Agentic Robot with Tool-use), a tool-injection fine-tuning framework that lets an existing VLA model decide on the fly when to activate external vision, affordance, and embodiment tools before producing continuous robot actions.

What This Paper Is About

End-to-end VLA models such as OpenVLA, ECoT, and π0 can produce precise continuous actions on the tasks they were trained on, but they degrade sharply when observations are noisy, poorly lit, or taken from shifted viewpoints, and adapting them typically requires costly post-training. Modular approaches wrap robot abilities in fixed tool sets, which decouples capabilities but limits them to simpler, handcrafted action outputs. ART tries to combine both: keep the end-to-end continuous action generator intact while giving the same model the ability to reason about and call external tools to clean up its inputs.

Key Contributions

1. A data generation method that integrates multi-modal tools into existing VLA datasets, producing long tool-usage chains without collecting new action data by hand.
2. ART (Agentic Robot with Tool-use), a tool-injection fine-tuning framework that preserves the pre-trained VLA action generation ability while adding new tool reasoning capability.
3. A non-destructive architecture and training recipe consisting of tool token injection into the vocabulary and a dynamically activated LoRA module, so tool reasoning and embodied action generation are optimized separately instead of interfering.
4. Empirical demonstration in the LIBERO simulation and on the real-world Astribot S1 robot that the method improves robustness to new scenarios, objects, and action instructions, and that tool reasoning learned on the AT dataset transfers to a new robotic scene.

Main Findings

• Simulation and real-world gains over mainstream baselines: On AT LIBERO, ART-FAST reaches 81% (vision), 62% (affordance), 82% (embodiment) and a 75% average success rate, compared with π0-FAST at 60%/12%/45% (39% average), π0 at 65%/15%/10% (30%), and OpenVLA at 20%/10%/7% (12%).
• Transfer to a real robot: On the Astribot S1, ART-FAST scores 70%/55%/70% for a 62% average, versus π0-FAST 40%/30%/60% (43%), π0 40%/30%/40% (37%), and OpenVLA 5%/5%/10% (6.7%). The paper states the abstract-level claim of a 20% higher success rate than mainstream baselines.
• Tool use beats end-to-end retraining with equal data: A post-trained FAST model fine-tuned on the same training data but without intermediate tool-invocation reasoning reaches 71% (vision), 61% (embodiment), 65% (affordance), below ART-FAST across all three categories.
• Robustness to visual corruption is the clearest separation from reasoning-only models: On the affordance portion of AT, ECoT scores 12% with vision corruption, 58% without it (36% average), while ART-FAST scores 72% and 62% (67% average). ART-FAST is only slightly better than ECoT on clean input.
• Low data dependency: The tool-use dataset contains 30K tool-use trajectories and action demonstrations, described as much smaller than those used by baseline methods, and fine-tuning used 1 epoch.
• Generalization of tool reasoning across datasets: Fine-tuning on the AT dataset, which draws on LIBERO, Bridge v2, and DROID, lets the model apply tools in new tasks without task-specific re-training.
• Baselines struggle below 30% average: OpenVLA and π0 are reported to average significantly below 30% success across the LIBERO simulated environment and the real-world Astribot S1 tasks.

Methodology in Plain English

The starting point is a pre-trained 3B π0-FAST model, which predicts chunks of continuous robot actions. ART changes the model's output space: alongside the usual continuous action, the model may emit (a) language reasoning tokens and (b) a discrete tool-use vector, where each entry says whether a given tool function is on or off. Because tool states are binary, they are represented as tokens using the last N entries of the vocabulary, following RT-2 and OpenVLA, so tool selection is trained with ordinary next-token prediction and cross-entropy loss.

Training is split so the two behaviors do not conflict. The VLA backbone is frozen and only LoRA modules dedicated to tool reasoning are trained for one epoch on the 30k-example AT dataset, on 8 × A800 GPUs with a batch size of 24, an initial learning rate of 5 × 10⁻⁵, and a 1k-step warm-up. At inference the model first reasons over the raw observation and activates tools with the LoRA module; after the tools enhance the observation, the LoRA outputs are masked out so the original backbone generates the embodied action. With an action-chunking scheme, this reasoning happens once per chunk of H timesteps rather than at every control step.

The tool set covers three input modalities. Visual enhancement includes 10 types of tools such as low-light enhancement, denoising, jitter correction, and deblurring. Affordance enhancement uses depth estimation and object detection. Embodiment enhancement covers camera rotation, zooming, and resetting the robot's bodily state.

The training data is built by extending existing VLA data rather than collecting new robot trajectories, using a three-step pipeline inspired by T3-Agent: task design, reasoning generation, and tool-trajectory synthesis. Existing tasks are "degraded" (random visual disturbances, harder language instructions that require affordance information, or simulated viewpoint and initial-state shifts), the appropriate tools are recorded into a reasoning chain, and a GPT model is prompted to chain the tools and reasoning into a full long-horizon trajectory. The resulting dataset is named AT (Action with Tool); the paper claims to be the first to introduce a VLA dataset with long-trajectory tool-use reasoning.

Why This Matters

Impact on research. The paper argues that tool use shrinks the effective action solution space relative to a whole continuous action space, which improves generalizability and reduces data dependency. It offers a route between the modular and end-to-end VLA camps: keep precise continuous control, but add plug-and-play tools instead of expensive post-training or full retraining, while avoiding the catastrophic forgetting that naive recipe coupling causes.

Real-world applications.

• Pick-and-place in low-light or dark conditions and at novel camera viewpoints, which the paper uses as its headline example.
• Affordance-heavy tasks such as identifying the "farthest object" or a target position, using depth estimation and object detection tools.
• Robots operating under embodiment shift, where camera rotation, zoom, or body-state reset can recover a workable observation.
• Deployment on dual-arm humanoid platforms such as the Astribot S1, a 16-degree-of-freedom robot trained here on a 16k pick-and-place dataset covering 80 objects and 10 container types.

Industry relevance. Because new capabilities arrive as swappable modules and the backbone stays frozen, the approach suggests lighter-weight deployment and incremental addition of tools without retraining the base policy, which matters for teams maintaining fleets of robots in messy, variable environments.

Future Directions

• Scaling and diversity of tool sets. The paper notes only 10 visual enhancement types plus a small affordance and embodiment set; how far tool libraries can grow before token and reasoning budgets become limiting is not established.
• Open-loop real-world evaluation. The setup section mentions an open-loop real-world testing setup alongside the closed-loop Astribot S1 scenario, but results for it are not reported, leaving that evaluation route open.
• Extending beyond explicit tool categories. The framework treats tools as modifications of observations; whether tools that alter the action space itself or perform multi-step physical sub-skills can be handled the same way is not addressed.
• Reliability of generated reasoning data. The AT pipeline relies on GPT models to generate tool chains and trajectories; the paper does not report how errors or hallucinations in that synthesis propagate into policy behavior, nor does it report failure cases or dataset statistics in the main text (these are deferred to an appendix).

Target Audience

Robotics and embodied-AI researchers working on VLA models, robot policy learning, and tool-augmented agents, particularly those interested in bridging modular and end-to-end paradigms. It is also relevant to engineers deploying manipulation policies on real hardware who need robustness to lighting, viewpoint, and sensor noise without full retraining. Readers need familiarity with VLA architectures, LoRA fine-tuning, and token-based action discretization to follow the formulation; the experimental tables are accessible to a broader audience.

Authors’ abstract

This paper integrates end-to-end Visual-Language-Action (VLA) models with agentic tool-use to propose Agentic Robot with Tool-use (ART). ART is a tool-injection framework that tunes any VLA model to leverage off-the-shelf tool modules for low-level vision, high-level affordance, and embodiment enhancement. Compared to vanilla VLA models with a whole continuous action solution space, ART reduces the complexity of the action solution space through tool-use, which not only improves generalizability across different tasks but also reduces data dependency. To demonstrate the advantages (high generalizability and low data dependency) of this framework, we first built a dataset of 30K tool-use trajectories and action demonstrations, which is much smaller than those used by baseline methods. We then designed a training regimen for long-trajectory tool-use reasoning in challenging environments. Experiments show that ART achieves a 20% higher success rate than mainstream baselines on simulation and real-world tasks, such as pick-and-place in the dark at novel viewpoints. Empirical results highlight the benefits of an agent-based approach: modular tool utilization enables more efficient training, lightweight deployment, and scalable integration of new tools. This design fosters robustness, adaptability, and extensibility, paving the way for the practical deployment of VLA systems in complex real-world scenarios.