Transferring the Intelligence of VLMs to Robotic Control

AI summary

Overview

Research area: Robotics — specifically robot manipulation, vision-language models (VLMs) as control agents, and generalization from simulated to real-world embodiments. The work sits at the intersection of embodied AI, agentic foundation models, and robotic action interfaces.

Technical level: Intermediate. The core idea is conceptually simple (let a VLM issue discrete, human-readable motion commands in a closed loop), but evaluating it requires familiarity with VLA (vision-language-action) baselines, benchmark protocols, in-context learning, and robot state representations.

Scope in one sentence: The paper introduces Robo Dawn, a human-intuitive discrete action interface plus an in-context learning scheme that lets a frozen pretrained VLM control robots without any task-specific robot training, and evaluates it on RoboTwin 2.0 C2R, RoboDojo, and real Franka and Piper robots.

What This Paper Is About

Most robotic control systems today are trained on large collections of robot data, an approach that is expensive, embodiment-specific, and — per the authors — risks degrading the general reasoning abilities of pretrained VLMs. This paper asks whether the general intelligence already present in a pretrained VLM can instead be transferred to physical robot control through a lightweight interface, rather than re-learned from robot data. The goal is to show that a frozen VLM, given a compact set of discrete motion commands and a small number of in-context demonstrations, can perform closed-loop manipulation competitively with policies trained on benchmark-specific robot data.

Key Contributions

1. Robo Dawn interface: A human-intuitive interface that exposes robotic control to an agentic VLM through a compact vocabulary of discrete motion primitives (translation, rotation, gripper, orientation presets, and utility commands), enabling closed-loop manipulation without task-specific robot training.

2. Interface-aligned in-context learning (ICL) scheme: A two-level demonstration context — a shared command primer (D_prim) plus task-level demonstrations (D_task) — that grounds a frozen VLM in both primitive action semantics and task-solving strategies, with no parameter updates.

3. Empirical SOTA on two simulation benchmarks: Robo Dawn reaches strong zero-shot performance and state-of-the-art results on RoboTwin 2.0 C2R and RoboDojo using only a single in-context demonstration, surpassing robot-trained and agentic baselines.

4. Real-world transfer and failure analysis: The same framework is deployed on real Franka and Piper robots, and the authors document three representative failure modes (precision limits, inverse-kinematics-related execution errors, and incorrect success judgments).

Main Findings

• Zero-shot beats trained policies on RoboTwin 2.0 C2R: Without any task-specific training, Robo Dawn with GPT-6 Astra achieves 53.2% success, above π0.5 (46.0%) and LingBot-VLA (50.4%). Other robot-trained baselines listed are lower: FastWAM 1.9%, StarVLA 10.6%, GalaxeaVLA 12.7%, Xiaomi-Robotics-0 18.2%, GR00T-1.7 20.7%, X-VLA 20.9%, X-WAM 25.8%.

• One in-context demonstration produces large gains: On RoboTwin 2.0 C2R, the GPT-6 Astra variant rises from 53.2% zero-shot to 73.6% one-shot, exceeding the best agentic VLA baseline, HarnessVLA (Claude Code), at 58.4% by 15.2 percentage points. With Gemini-3.8-Flash the same benchmark goes from 47.0% zero-shot to 62.2% one-shot.

• RoboDojo improvements are consistent: Success rate improves from 35.67% zero-shot to 47.17% one-shot (scores 39.92 and 54.63 respectively), well above the full-set post-trained DM0.5 at 19.34% success (score 24.90). On the 34-task subset excluding 8 open tasks, Robo Dawn obtains 33.96% zero-shot and 43.33% one-shot (scores 39.02 and 51.29).

• Test-time scaling: On RoboDojo, success increases steadily with the per-episode command budget: the one-shot setting goes from 31.2% at 60 commands to 47.2% at 240 commands, while zero-shot goes from 23.7% to 35.7%. The authors attribute this to failure-recovery capability and the memory mechanism.

• More demonstrations help, but plateau and then dip: On RoboTwin 2.0 with Gemini-3.8-Flash, success rises 47.0% (0 shots) → 62.2% (1) → 63.6% (2) → 65.4% (4), then falls to 62.7% at 8 shots, which the authors associate with long-context interference.

• Performance scales with VLM strength: Under the same one-shot setting, success goes from 14.4% (GPT-5.6-Luna) to 43.2% (GPT-5.6-Sol), 45.0% (Seed-2.1-Pro), 62.2% (Gemini-3.8-Flash), and 73.6% (GPT-6 Astra).

• Harness ablations show reasoning and spatial grounding matter most: Zero-shot with Gemini-3.8-Flash, removing reasoning drops success from 47.0% to 34.8%; removing grid-based localization drops it to 32.4%; removing the command primer drops it to 44.0%.

• Real-world results are mixed and task-dependent: With Gemini-3.8-Flash in the zero-shot setting, block-in-basket on Franka succeeds 9/10, block stacking on Franka 5/10, and cloth folding on Piper 0/10. The authors note cloth folding involves substantial end-effector rotation and orientation adjustment, which is less represented in web-scale pretraining data, and that several near-successful cloth-folding trials were not counted because the final folds were not sufficiently neat.

• Efficiency trade-off is real: Using Seed-2.1-Pro for latency evaluation, Robo Dawn shows 9.74 s inference latency, 3.4 commands, 2.09 s motion execution time, and an inference-to-motion ratio of 4.65 — slower than π0.5 (101 ms, 45.0 steps, 2.70 s, ratio 0.037), StarVLA (70 ms, 16.0 steps, 0.96 s, 0.073), X-VLA (143 ms, 28.6 steps, 1.71 s, 0.084), FastWAM (523 ms, 27.3 steps, 1.64 s, 0.32), Motus (1.93 s, 16.0 steps, 0.96 s, 2.01), and LingBot-VA (8.89 s, 22.2 steps, 1.33 s, 6.67). The paper notes that an inference-to-motion ratio below 0.5 would permit streaming inference, which Robo Dawn does not achieve.

• Three failure modes identified: insufficient manipulation precision at high-accuracy stages such as insertion; inverse-kinematics-related execution errors where semantically reasonable commands collide with nearby objects; and incorrect success judgment where the model stops before the task's success criterion is met.

Methodology in Plain English

The interface. Instead of predicting continuous joint-level controls, Robo Dawn gives the VLM a small grammar of discrete commands: move an arm along an axis by a distance, rotate it about an axis by an angle, point the gripper to a preset orientation, open or close the gripper, or issue home, wait, and done. Translation distances are clipped to 20 cm per command and rotations to 90 degrees per command. All spatial commands are defined relative to the gripper interaction point (GIP) — the midpoint between the two fingertips — rather than the wrist pose, so that annotations, state reporting, and commands all refer to the same physical point.

The loop. At each decision round, the VLM receives annotated visual observations, measured robot state, execution feedback from the previous round, and interaction memory, plus two fixed pieces of context: a robot-environment profile describing interface conventions (workspace constraints, camera, grid-based localization, gripper properties) and the in-context demonstration set. The VLM outputs semantic action commands together with a structured response containing a task-progress estimate, the current plan, and a compact scratchpad. Those commands are grounded into planned robot motions, executed until the robot is stationary, and the resulting outcomes are converted into feedback and memory updates for the next round. The VLM's parameters stay frozen throughout; all adaptation comes from observations, feedback, and memory.

The demonstrations. Raw expert trajectories are not directly usable because they consist of continuous low-level actions. The authors re-express each trajectory in the semantic command space by reducing it to end-effector waypoints and gripper states, then reaching each waypoint with a short sequence of translation, rotation, and gripper commands. Demonstration context is split into a shared command primer (how each primitive behaves) and task-level demonstrations (when primitives are composed into task-solving behavior). With zero, one, or more task demonstrations, the setup corresponds to zero-shot, one-shot, or few-shot. Demonstrations are collected in scenes disjoint from evaluation — from the benchmark's scripted expert in simulation and from human teleoperation on real robots — and in simulation these are the same trajectories used to train the compared robot policies. The in-context image budget is capped at 16 observations per round; when exceeded, images are kept only for semantically informative rounds such as grasping, rotation, and task completion.

The evaluation. Two simulation benchmarks are used: RoboTwin 2.0 C2R, with 50 bimanual manipulation tasks, where baselines are jointly post-trained on 50 clean demonstrations per task and evaluated under domain randomization, with 10 independent evaluation runs per task; and RoboDojo, with 42 tasks on a bimanual platform, reported over 5 runs and evaluated using both success rate and task progress score. Real-world deployment uses Gemini 3.8 Flash on Franka and Piper robots in the zero-shot setting.

Why This Matters

Impact on research. The paper argues for a complementary direction to the current focus on scaling robot datasets and training action models: for a broad class of manipulation problems, a strong pretrained VLM can serve as the high-level decision-making engine, with a simple interface and a few demonstrations transferring its intelligence to control. It also cites evidence that training VLMs for action prediction may degrade general-purpose capabilities such as instruction following and reasoning, and notes that robot data remain orders of magnitude scarcer than the language-vision data used to build modern VLMs — making the parameter-free approach attractive for preserving general intelligence. The work is closely related to concurrent work on Show-Harness, which also shows frontier VLMs performing closed-loop control through a compact semantic action interface; the authors position Robo Dawn as going further by co-designing an effective ICL strategy.

Real-world applications:

• Pick-and-place and sorting in logistics or warehouse settings, the class of task closest to the reported block-in-basket result (9/10 on Franka).
• Simple assembly and stacking tasks where precise final alignment matters more than the grasping approach (block stacking succeeded 5/10, with precision identified as the limiting factor).
• Deformable-object handling such as cloth folding, though the paper reports 0/10 success and identifies end-effector rotation as the key difficulty, so this is currently not a viable deployment target.
• Rapid prototyping of new manipulation tasks, since no task-specific training data collection or parameter updates are required — only a handful of demonstrations.

Industry relevance. The approach removes the need to collect large task-specific robot datasets and to retrain models for each new task or embodiment, provided a sufficiently capable VLM is available. The paper's scaling results show performance tracking VLM capability directly, meaning robotics performance improves as general-purpose models improve. The main barrier to industrial use is inference speed: Robo Dawn's 9.74 s latency and 4.65 inference-to-motion ratio make it unsuitable for high-frequency control, and the authors explicitly list safety in deployment as a concern given that erroneous actions may cause collisions, object damage, or risks to humans.

Future Directions

• Improve the action interface for rotation and fine-grained precision. The ablations and failure analysis point to end-effector rotation and high-precision contact as the weakest areas, both because rotational motions may be less represented in web-scale pretraining data and because the discrete interface can be too coarse for subtle corrections.

• Reduce inference latency. Robo Dawn is slower than end-to-end VLA and WAM policies due to iterative VLM reasoning and closed-loop interaction. Closing the gap to an inference-to-motion ratio below 0.5 would enable pipelined or streaming execution.

• Extend beyond the discrete-granularity ceiling. Since test-time scaling shows that more commands yield better success, a natural question is whether adaptive granularity or hybrid low-level controllers could handle the precision failures the paper documents.

• Address safety and robustness in physical deployment. The paper flags safety as increasingly critical as systems approach general-purpose use; collision-free execution and reliable success criteria (given the observed mismatch between the model's notion of completion and benchmark criteria) remain open problems. The paper does not report any safety evaluation methodology.

Target Audience

Researchers and engineers working on robot manipulation, embodied AI, and vision-language-action models will find the core results most directly useful, particularly those interested in alternatives to large-scale robot-data training. The paper is also relevant to practitioners who want to prototype manipulation tasks quickly with existing VLMs, to VLM researchers interested in how general-purpose capabilities transfer to physical control, and to readers tracking benchmark progress on RoboTwin 2.0 and RoboDojo. Readers without robotics background can follow the main argument but will need familiarity with VLA baselines, inverse kinematics, and domain-randomized evaluation to interpret the numbers.

Authors’ abstract

Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control? We investigate this question through RoboDawn, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands. Using this interface, the VLM controls a robot in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions to the resulting state. Furthermore, we introduce an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies. Experiments on RoboTwin 2.0 C2R and RoboDojo demonstrate that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, RoboDawn outperforms several strong policies trained on benchmarkspecific robot data, while a single in-context demonstration further yields substantial performance gains and establishes state-of-the-art (SOTA) results. On RoboTwin 2.0 C2R, the success rate increases from 53.2% zero-shot to 73.6% one-shot, exceeding the solid baseline π0.5 (46.0%). Similar gains are observed on RoboDojo, where success rate improves from 35.67% zero-shot to 47.17% one-shot. The same framework also transfers to real-world robots, performing block-in-basket and block stacking on Franka.