Democratizing Robot-Use Agents: A Unified Harness with Open Models

AI summary

Overview

Research area: Robotics + AI agents (cs.RO) — specifically "robot-use agents" (RUAs), where a general-purpose multimodal language model drives a robot through tool calls rather than through a robot-specific policy network.

Technical level: Intermediate. The core ideas are graspable without robotics mathematics, but the evaluation involves simulation benchmarks, VLA/WAM baselines, and system-engineering trade-offs.

Scope (one sentence): The paper introduces uni-robot-agent, an open, modular harness that lets open-weight models (Qwen, DeepSeek, GLM) operate simulated and real robots without privileged simulator information, and shows that one such model beats a published proprietary-model result on the RoboDojo benchmark.

What This Paper Is About

Frontier proprietary models such as GPT-6 Astra and Opus 5.5 have shown surprising zero-shot manipulation ability, but because their weights are closed, researchers can only use them as-is — they cannot train them further, and API costs plus inference latency block large-scale experimentation. The paper asks whether open-weight models can close this gap if given a well-designed harness. The goal is to provide a reusable foundation (unified tools, efficient execution, realistic inputs, modular components) so that open models can be both competitive today and trainable on robot experience tomorrow.

Key Contributions

1. A unified, modular harness (uni-robot-agent) organized into five layers (agent, orchestration, tool, service, backend) with shared tool interfaces across models, embodiments, tasks, and environments. Shared tools are written once and reused across M environment adapters, avoiding N × M tool–environment implementations, and the resulting interaction records can be combined for later training.

2. An execution pipeline that overlaps model reasoning with physical action. Background tool calls return job handles, calls that share an exclusive resource execute in submission order, and a flush_queue operation cancels calls that have not yet started. The paper reports this overlap explicitly rather than treating it as an implementation detail.

3. A "no privileged inputs" operating regime plus six concrete design choices for making agent–robot interaction reliable: immutable observation snapshots, duplicate-request rejection after communication failures, combined motion-and-gripper commands, progress-checked motion with stopping criteria, numerical plus visual outcome feedback, and a sliding-window visual history with batched image eviction for cache reuse (with history compaction using the codex CLI compaction prompt).

4. An empirical decomposition of what makes robot use work, via comparisons across model generations, families, and sizes, tracing failures to four capability areas — general visual understanding, spatial grounding, geometric coding, and embodied reasoning — and correlating RoboDojo performance with MMMU-Pro, ERQA, ScreenSpot-Pro, and LiveCodeBench v6.

Main Findings

• An open-weight model beats a published proprietary result on RoboDojo. Out of the box, Qwen3.8-Flash-Next with uni-robot-agent reaches 25.00% success rate and a 31.39 Score, versus the published GPT-6 Astra–RoboProbe result of 22.48% SR / 28.97 Score. The authors state this is, to their knowledge, the first system to outperform closed-weight foundation models on RoboDojo using open-weight models without privileged information or distillation.

• A second open-weight family also clears the VLA/WAM baselines. DeepSeek-V4.1-Flash reaches 21.39% SR / 27.11 Score, above the listed π0.5 (6.93/11.44), Hy-Embodied-0.5-VLA (8.80/13.07), OpenWAM-α (11.92/17.18), and Xiaomi-Robotics-1 (13.93/20.07).

• The harness itself is a large part of the result. Running the same Qwen3.8-Flash-Next model on other harnesses collapses performance: RoboProbe gives 0.83/1.90, RPent 4.31/7.75, and GAP (Hybrid) 16.11/21.01 — a loss of 8.89% SR and 10.38 Score against the strongest baseline. The harness also helps closed models: GPT-6 Astra with uni-robot-agent averages 46.67/51.90, far above its RoboProbe result, so the design choices are not open-model-specific.

• Results are uneven across capability dimensions. Qwen3.8-Flash-Next is strong on memory (52.78/54.72) and open-vocabulary instructions (45.83/46.67) but weak on precision (2.08/10.21) and long-horizon tasks (6.25/23.75), where Xiaomi-Robotics-1 (23.67/38.39) and OpenWAM-α (25.33/34.93) do better.

• On 19 real-world tabletop tasks, Qwen3.8-Flash-Next scores 49.50. That edges fine-tuned π0.5 (47.75) and beats zero-shot Dream-1 (35.50), while GPT-6 Astra with the harness reaches 75.50. The paper notes the agents additionally receive depth from Intel RealSense D405 cameras whereas π0.5 and Dream-1 use RGB only. Per-dimension, π0.5 leads on object generalization (12.50 vs 6.00) while the agents lead on compositional generalization (GPT-6 Astra 20.00, Qwen 13.00, Dream-1 10.00, π0.5 6.00) and unseen tasks (8.00 for both agents vs 4.00 for both baselines).

• Robot use predates the newest models, but is uneven across families. GPT-5.6 Sol already achieves 19.17% SR / 24.04 Score and GPT-6.1 Sol reaches 37.50% SR / 45.63 Score. For Qwen, verified success appears only in the 3.8 generation: Qwen3.5–3.7 Flash, Qwen3.6–3.7 Plus, and Qwen3.5-122B record zero SR, while Qwen3.8-27B reaches 8.06% and Qwen3.8-Flash-Next 25.00%. GLM-5.3 Flash records 0%.

• Size matters less than training distribution. Qwen3.8-27B succeeds where Qwen3.5-122B and Qwen3.6/3.7 Plus (more than 150B parameters) all record zero SR; within Qwen3.8, Flash-Next (125B) outperforms the 27B model.

• Four failure modes track task performance. General visual misunderstanding occurs in 12.2–23.8% of trials for earlier Qwen Flash/Plus models, 3.2% for Qwen3.8-27B, and none for Qwen3.8-Flash or GPT-5.6 Sol. Geometry-coding errors affect 47.3–81.8% of executed calculations for earlier Flash/Plus cohorts versus 8.2% (Qwen3.8-Flash), 1.5% (DeepSeek), 0.7% (GPT-5.6 Sol), and none for GPT-6.1 Sol. Spatial grounding remains hard even for the best model: GPT-6.1 Sol has the lowest such failure rate at 31.5%. GLM's embodied-reasoning failures (repeatedly failing to reorient its wrist) produce a 26.2% error rate on the pick-and-place subset.

• General benchmarks correlate with robot-use performance. RoboDojo Score correlates positively with MMMU-Pro, ERQA, ScreenSpot-Pro, and LiveCodeBench v6 across six models, with the strongest correlations for embodied reasoning and general visual understanding and a weaker one for screen-based spatial grounding.

• Privileged depth is not necessary if camera parameters are available. Removing depth changes GPT-6 Astra from 55.83 to 55.04 (98.6% retained) and Qwen from 35.89 to 31.39 (87.5% retained). Removing camera parameters too drops Qwen to 9.13 and Astra to 31.29.

• The skill library is not load-bearing. Disabling robot skills and skill writing yields 30.31 Score and 24.58% SR, retaining 96.6% of the skill-enabled score, an SR gap of 0.42 percentage points.

• Cost gap is roughly an order of magnitude. Qwen reaches 31.39 Score at $1.09 per episode; Astra reaches 51.90 at $21.29 per episode, nearly 20x more.

• Open models progress more slowly but pipelining helps. To reach a Score of 25, Astra needs 7.4M tokens per episode versus 23.5M for Qwen and 49.9M for DeepSeek, taking 42.6, 95.5, and 99.2 minutes respectively. Pipelined execution shortens episodes on all five tested tasks by 4.8% to 73.7% (e.g., swap_T 12.5 vs 34.7 minutes; press_by_number 13.7 vs 25.0 minutes), with 84.5%–93.8% of physical execution time overlapping model requests, versus zero in serial runs.

Methodology in Plain English

The team built a software layer that sits between a language model and a robot, similar in spirit to how a coding assistant uses terminal tools. The model is given a fixed set of named operations — move the gripper, look at the scene, run a skill, control the episode — with the same arguments, units, and feedback format whether it is talking to a simulator or a physical arm. An adapter per environment translates those shared operations into native commands, so adding a robot means writing one adapter rather than reimplementing every tool.

To keep the model from stalling, the harness lets it fire off a move in the background and keep reasoning, while moves that touch the same physical resource are forced into a queue. To keep the setup honest, the model sees only what a real deployment would offer: RGB images, camera calibration, robot proprioception and descriptions, and the task instruction — no simulator ground-truth depth, no CAD models, no task-verifier success signals, no web access. The authors manually checked that the models never made invalid access to prohibited information.

Around that skeleton they made six practical fixes for physical reality: binding images, calibration, and robot state into one immutable snapshot so a pixel is never interpreted with a newer camera pose; recording request identities so a missing reply does not cause a duplicated physical action; letting one command set both target pose and gripper opening; generating intermediate waypoints and checking progress so motion stops on arrival or when stalled; returning numerical pose errors and stopping reasons alongside outcome images; and replacing stale images with text placeholders in batches so the prompt prefix stays cacheable.

The evaluation has two halves. In simulation they use RoboDojo's 42 tasks across five dimensions — generalization, precision, long-horizon, memory, and open-vocabulary — with three seeds and two layouts (six episodes per task, 252 episodes total), reporting success rate and a partial-credit Score on a 0–100 scale and averaging per-dimension results following the official protocol. In the real world they built an in-house benchmark of 19 tasks across five equally weighted dimensions scored 20 points each, on a shared dual-arm platform. They then ran the same harness over older and newer generations, different families, and different model sizes, hand-analyzed failure trajectories to categorize errors, and correlated benchmark scores with general capability benchmarks.

Why This Matters

Impact on research. The central blocker in robot-use-agent research was that the best models were closed: you could design a harness around them, but never fine-tune them, cheaply cache their inference, or run large-scale interaction studies. By showing an open-weight model can match or beat a published proprietary result while costing about $1.09 per episode instead of $21.29, this paper makes RUA training a tractable research program. The failure-mode analysis also reframes the agenda: instead of fitting action outputs, future training should target visual understanding, spatial grounding, geometric coding, and embodied reasoning jointly.

Real-world applications:

• Affordable local deployment of manipulation agents where per-call API pricing or data-egress rules make proprietary models impractical.
• Warehouse and lab automation built on a harness where swapping the arm or the camera only requires a new adapter.
• Data collection: the unified interaction records are explicitly designed to be pooled across environments as training data, which supports continuous improvement from deployed fleets.
• Home or service robots that rely on ordinary RGB cameras and known calibration rather than calibrated depth sensors, since removing depth cost only 1.6% of score for the strongest model.

Industry relevance. The paper gives a concrete engineering blueprint for latency and cost: reasoning–execution overlap bought 4.8%–73.7% shorter episodes, and the modular layer boundaries mean a model vendor can be swapped without rebuilding robot integration. For anyone building agent infrastructure rather than robot policies, the six design choices (snapshot consistency, duplicate suppression, combined motion-plus-gripper commands, progress-checked motion, diagnosable feedback, cache-friendly image eviction) are directly transferable lessons about acting in a world that changes while commands are in flight.

Future Directions

• Joint capability training. The paper argues that RUA training should develop visual understanding, spatial grounding, geometric coding, and embodied reasoning together, for example by constructing robot-data training tasks that target each capability — but it does not demonstrate such training, which remains the obvious next step given the harness was built to support it.

• Precision and long-horizon execution. These are the weakest measured dimensions for the open model (precision 2.08/10.21, long-horizon 6.25/23.75), and the conclusion names them as remaining challenges. Whether they are a harness problem or a model-capability problem is unresolved.

• Spatial grounding as the persistent bottleneck. Even the best-evaluated model, GPT-6.1 Sol, still shows a 31.5% spatial-grounding failure rate, and the paper notes that ScreenSpot-Pro correlates only weakly because it tests 2D screen grounding rather than the 3D grounding manipulation needs. A better measurement and training signal for 3D correspondence is an open question.

• Broader embodied reasoning beyond grasping. The embodied-reasoning analysis is deliberately narrow — restricted to pick-and-place grasp-pose validity judged by GPT-6 Astra — and the authors note GPT-6 Astra still struggles with bimanual coordination. How to evaluate the rest of embodied reasoning at scale is left open.

Target Audience

Researchers and engineers working on embodied agents, robot-learning infrastructure, or tool-using multimodal agents — especially those who want to train or fine-tune robot-use models rather than only prompt proprietary APIs. It is also useful for practitioners who care about the systems side (latency, caching, cost, failure recovery) of agents acting in the physical world, and for robotics teams deciding whether an agentic, open-weight stack can substitute for a fine-tuned vision-language-action policy. Readers without any robotics background will follow the argument but will need the appendix for task-level detail, which is truncated in the provided content.

Authors’ abstract

Robot-use Agents (RUAs) have shown promising zero-shot manipulation capabilities, but reliance on frontier, proprietary models limits research into further training. To democratize RUA research, we introduce uni-robot-agent, a unified harness for competitive open-weight robot use that supports further training. Without privileged inputs, an out-of-the-box Qwen3.8-Flash-Next achieves a RoboDojo Score of 31.39, surpassing the published GPT-6 Astra-RoboProbe result of 28.97 and many VLA/WAM baselines, and performs competitively on 19 real-world tasks. Our analysis across model generations, families, and sizes links effective robot use to four general capabilities, suggesting that future RUA training should develop them jointly.