Show-Harness: Just a VLM Agent Can Play Robots

AI summary

Overview

Research area: Robotics and embodied AI, specifically vision-language-model (VLM) driven robot manipulation.

Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with VLA models, robot end-effector control, and fine-tuning concepts.

Scope: The paper proposes a semantic action interface ("Embodied Harness") that lets both frontier and small open-source VLMs control real robot arms directly, without embodiment-specific policy pretraining.

What This Paper Is About

Foundation VLMs already "know" a great deal about objects, spatial relations, and long-horizon tasks, but that knowledge rarely turns into robot motion. Existing approaches either fine-tune VLMs into opaque continuous-action regressors (VLA models) that must be re-adapted per task and robot, or keep the VLM at a high level and hand physical control to a hidden downstream controller. Show-Harness takes a middle path: it exposes a small vocabulary of discrete, semantically named action units (move forward, rotate clockwise about z, grasp, release, done) that the VLM reasons over natively, while a deterministic per-robot interpreter converts each unit into a bounded physical motion. The goal is to make the VLM directly responsible for fine-grained physical decisions while remaining reusable across robots.

Key Contributions

1. Show-Harness, a model-agnostic embodied harness that connects foundation VLMs to robots through a compact semantic action interface — discrete, incremental, embodiment-agnostic units that are deterministically grounded into physical motion by a per-embodiment interpreter.
2. Two deployment modes on one interface: zero-shot control with closed-source frontier VLMs (no fine-tuning) and low-cost adaptation of small open-source VLMs (2B scale) via LoRA fine-tuning over just a few GPU-hours, predicting actions through the model's native vocabulary without special action heads or tokens.
3. Strong empirical generalization across tasks, environments, and embodiments, including sim-to-real transfer and cross-embodiment transfer between a Franka arm and a bimanual AgileX rig, outperforming representative VLA, VLA-centric agent, and code-as-policy baselines.
4. GUMI, a GUI-based manipulation interface that exposes the same semantic action space to humans, computer-use agents, and VLM agents — enabling cross-embodiment demonstration collection without specialized teleoperation hardware, plus human-agent collaborative rollout correction.

Main Findings

• Same interface, two capability regimes: A frontier VLM (Gemini-3.1 Pro) controls robots zero-shot, while a 2B open-source model (Qwen3.5-2B) with LoRA on ~3% of parameters matches or beats trainable VLA baselines. Both modes consistently outperform π0.5, GR00T, Harness VLA, Goal-VLA, CaP-X, and RATS across task, environment, and embodiment shifts.
• Sim-to-real transfer succeeds where VLAs fail: The fine-tuned policy trained only on 230 simulated episodes (ManiSkill + RoboLab, collected through the same GUMI interface) transfers to the real Franka, while trainable VLA baselines fail under the same setup.
• Precision adapts through the interpreter, not relearning: Reducing the interpreter step from 2 cm to 1 cm on stacking and peg insertion lifts zero-shot success from 60% to 82% and fine-tuned from 40% to 65%, without changing the VLM interface or retraining. π0.5 reaches only 18% without extra fine-grained training.
• Compositional extrapolation: Incremental 15° rotation units let the fine-tuned model reach 70% on an unseen 90° carrot orientation (trained only on 0° and 45°), versus 20% for π0.5. Similarly, composing two orthogonal translations into a diagonal displacement cuts steps with little success loss.
• Robustness to workspace expansion: On nested workspace regions from the core 25% to 90% near the boundary, Show-Harness degrades only mildly while π0.5 drops sharply.
• Semantic adaptability is preserved: On reasoning-heavy tasks (locating a block under one of three cups; arranging letters into "SHOW"), zero-shot with Situated Planning reaches 85%, while fine-tuned alone hits 10% and π0.5 reaches 0%. Given frontier-generated subtask instructions, fine-tuned jumps to 70%, while π0.5 stays at 5%.
• Visual in-context learning works: With a single human or robot video demonstration, the zero-shot agent follows the demonstrated object order in 20/20 trials; without a demonstration it succeeds only 20% of the time.
• Multi-arm coordination is explicit: Jointly predicting both arms' actions improves success on table-tidying and banana-handover tasks and eliminates collisions, compared to two independent single-arm agents.
• Conventions, not names, do the grounding: An ablation swapping the six translation units' representation shows that arbitrary symbols with written physical conventions nearly match the default semantic names, while symbols alone succeed in only 1/20 episodes. Semantic names act mainly as a useful prior.
• Plugin value is task-dependent: Multi-View Guidance, Proprioception, Subtask Planning, Action Chunking, Adaptive Step, Action History, and Failure Recovery all matter on standard tasks. Visual Prompt and Situated Planning have no effect on routine pick-and-place but raise success from 40% to 85% on handle-aware grasping and from 35% to 85% on hidden-object search, respectively — justifying their default-off design.
• More thinking is not always better: Stronger frontier VLMs improve zero-shot success, but raising the inference-time reasoning budget mainly reduces redundant steps with little success gain and can cost up to 3.4x wall-clock time. Errors concentrate on fine-grained grasping and placement, not planning, and all models produced valid action units in over 98% of responses.
• 2B is a sweet spot for fine-tuning: Larger backbones help mainly on fine-grained tasks; 1B-scale models over-correct locally near targets and produce longer episodes, though smaller models can beat larger ones on moving-object tasks like the tennis ball.

Methodology in Plain English

The researchers split robot control into two layers. The VLM only ever sees and emits a tiny vocabulary of semantic symbols — six translation directions defined relative to whatever camera view is being shown, incremental rotations about x, y, or z, plus grasp, release, and done. Directions are anchored to observable views, so the model's spatial reasoning maps directly onto actions. A separate, embodiment-specific interpreter then translates each symbol into a small bounded motion: it updates a 6-DoF Cartesian pose setpoint by a calibrated translation or rotation increment, projects it into the robot's workspace, and hands it to whatever low-level controller that arm uses (impedance control for the Franka, inverse kinematics plus streamed joint targets for the AgileX, operational-space commands in simulation). Gripper commands bypass the pose update.

Around this interface sits the "harness," a perceive–reason–act loop with configurable plugins. Perception plugins turn multiple camera views and robot proprioception into text the VLM can read. Reasoning plugins decompose instructions into subtasks with visually checkable completion criteria, defer uncertain decisions until evidence appears, chunk actions to cut query frequency, adapt step size to target distance, and highlight task-relevant visual targets. Action plugins keep a short history of recent actions to prevent oscillation and detect grasp failures to trigger recovery.

Because the action space is discrete and human-operable, the team built GUMI, a keyboard/GUI front end where each semantic unit maps to a labeled control. Humans and agents "play" the robot through the same buttons, and every step is logged as a ready-to-train (observation, action) pair. For fine-tuning, they collect 164 real-robot episodes (7.8K decision steps) across Franka and AgileX plus 230 simulated episodes (13.5K steps), then train a 2B open-source VLM with rank-64 LoRA on language-model linear layers only, minimizing token-level cross-entropy on the target action symbol in the model's native vocabulary.

Why This Matters

Impact on research. The paper reframes the VLM-to-robot gap as an interface design problem rather than a model capacity problem. If a compact semantic action space can unlock zero-shot control from frontier models and let 2B models compete with much larger VLA pipelines, then embodiment-specific action pretraining — currently one of the field's largest costs — may be largely avoidable. The demonstration that written conventions alone supply most of the grounding (arbitrary symbols with conventions nearly match semantic names) is a concrete design lesson for anyone building agentic robot systems, as is the finding that planning is not the bottleneck; fine-grained grasping and placement are.

Real-world applications.

• Warehouse and logistics picking, where the same semantic policy can be redeployed to new arm hardware by swapping only the interpreter.
• Household assistance tasks such as tidying cluttered tables, arranging objects into semantic configurations, and bimanual drawer opening.
• Handling of deformable and irregular objects (teddy bears, bananas, rolling balls) where rigid-object trajectory regression tends to struggle.
• Low-cost remote demonstration collection: GUMI lets operators label robot data from a browser keyboard without owning or colocating with a teleoperation rig.

Industry relevance. Robot foundation-model companies spend heavily on embodiment-specific pretraining and teleoperation hardware. A semantic interface that inherits each new frontier VLM release for free, supports cheap 2B deployments on a single consumer GPU, and records reusable cross-embodiment data directly attacks all three cost centers. The human-readable action vocabulary also creates a natural path to human-in-the-loop correction during agent rollouts, which matters for safety-critical deployment.

Future Directions

• Scalability of demonstration data. Only 164 real episodes and 230 simulated episodes were used. How the two modes scale with thousands of episodes, and whether the fine-tuned mode eventually overtakes zero-shot frontier control, is untested.
• Automatic interpreter construction. Adapting to a new robot currently requires hand-writing an interpreter with workspace limits, motion-frame mappings, and step calibrations. Learning or auto-calibrating interpreters from a handful of trials would remove the last embodiment-specific engineering step.
• Safety and failure semantics at scale. The paper relies on workspace projection and grasp-failure recovery, but does not characterize what happens when the VLM emits semantically valid yet physically unsafe sequences, or how guarantees could be enforced inside the harness.
• Beyond parallel-jaw pick-and-place. Dexterous hands, mobile bases, and long-horizon multi-stage tasks are largely unexplored; the rotation-extrapolation result hints that incremental units may generalize to dexterous reorientation, but this is untested.

Target Audience

Robotics and embodied-AI researchers working on VLM-driven manipulation, especially those frustrated by the cost of embodiment-specific VLA pretraining. Also valuable for applied ML engineers shipping robot products who need to choose between fine-tuning a large VLA and fine-tuning a small VLM behind a semantic interface, and for HCI-adjacent researchers interested in GUI-style robot teleoperation and human-agent collaborative data collection. Readers without prior exposure to VLA models or robot kinematics will find the action interface easy to grasp but the experimental comparisons harder to contextualize.

Authors’ abstract

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.