XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment

AI summary

Overview

Research area: Robotics — robot learning infrastructure, policy evaluation, and deployment tooling (vision-language-action models, world-action models, diffusion visuomotor policies, imitation-learning baselines).

Technical level: Intermediate. The paper is written as a systems contribution rather than an algorithmic one; readers need some familiarity with robot policy evaluation benchmarks and software packaging, but no deep learning theory.

Scope: XPolicyLab is a standard plus open ecosystem that defines one contract between robot policies and evaluation environments — a minimal adapter interface, shared observation/action schemas, and dependency-isolated local or remote serving — and demonstrates it with 42 integrated policies across RoboTwin, RoboDojo simulation, and RoboDojo-RealEval.

What This Paper Is About

Robot policies each ship their own software dependencies, observation formats, action conventions, and runtime interfaces, so getting one policy to run in a new benchmark, simulator, or physical robot means writing new platform-specific code every time. Stated as a systems problem, connecting N policies to M environments costs O(N·M) separate integrations. The paper introduces XPolicyLab, a unified standard and open ecosystem that reduces this to O(N+M) by giving policies and environments one shared interface, while letting each side keep its native software stack.

Key Contributions

1. A unified policy–environment contract. XPolicyLab standardizes observation and action representations, action prediction, batched execution, and episode management through a minimal adapter interface, while leaving each policy's architecture, dependencies, checkpoints, and training procedure untouched.

2. Dependency-isolated serving architecture. Policy inference is separated from environment execution so policy servers and environment clients can run in independent software environments, on the same machine or across a network, using a WebSocket protocol with MessagePack serialization extended for arrays.

3. Machine-readable agent skills for conformance. The integration procedure is packaged as an integration skill and an audit skill, so scaffolding, implementing, and auditing a new adapter becomes a guided, checkable procedure that coding agents (Cursor, Claude Code, Codex) can execute under human supervision.

4. An open ecosystem with deployment evidence. 42 robot policies are integrated as of August 8, 2026, spanning VLA models, world-action models, diffusion-based visuomotor policies, memory-augmented policies, and conventional imitation-learning baselines, with deployments on RoboTwin and RoboDojo (including RoboDojo-RealEval).

Main Findings

• Order-of-magnitude reduction in integration cost. In a controlled study with N=6 engineers unfamiliar with XPolicyLab connecting π0.5 to RoboDojo simulation, median total integration effort fell from over 5 h (from scratch) to roughly 2 h (manual, via the standard) to roughly 30 min (with agent skills). Hand-written lines of code went from roughly 300 to roughly 120 to roughly 0.

• Per-phase breakdown of that effort. Median environment setup was 2 h from scratch, 30 min manual, 10 min with skills; observation/action glue code was 2 h, 1 h, and 10 min; debugging to first rollout was 1.5 h, 30 min, and 10 min.

• Reproduction is cheaper than integration. For a policy that is already integrated, installation and a single evaluation command complete in approximately 10 minutes, excluding the bandwidth-bound checkpoint download.

• Heterogeneity is confined to the policy side. Across the 42 adapters, model-specific code varies by an order of magnitude while the environment-facing loop stays within a few lines of a fixed reference — evidence that the contract, not ad hoc tolerance, absorbs the variation.

• One adapter serves multiple settings. The same adapters drive RoboTwin simulation, RoboDojo simulation, and RoboDojo-RealEval across three bimanual embodiments (ARX X5, Piper, Piper X), with policy-specific inference logic remaining behind the serving boundary.

• RoboTwin top-10 clean vs. randomized success rates (snapshot Aug 10, 2026): FastWAM 77.8 / 1.9; Spatial Forcing 77.2 / 9.5; π0.5 70.7 / 46.0; X-WAM 70.0 / 25.8; X-VLA 68.0 / 20.9; EventVLA 65.6 / 15.7; AHA-WAM 64.3 / 3.2; Xiaomi-Robotics-0 62.9 / 18.2; G0 62.7 / 9.1; Abot-M0 57.4 / 22.9. All listed policies completed evaluation on all 50 tasks under both settings.

• RoboDojo simulation top-10 by average score, with success rate (snapshot Aug 4, 2026): G0.5 20.23 / 14.88; Xiaomi-Robotics-1 20.07 / 13.93; Hy-Embodied-0.5-VLA 13.07 / 8.80; Spatial Forcing 12.38 / 8.04; π0.5 11.41 / 6.91; InternVLA-A1.5 11.15 / 7.14; VLAct 10.66 / 7.60; X-VLA 10.13 / 6.52; X-WAM 7.69 / 3.83; Xiaomi-Robotics-0 6.93 / 4.18. Scores are averaged over the five capability dimensions (Generalization, Precision, Long-Horizon, Memory, Open) across 42 simulation tasks.

• RoboDojo-RealEval top-10 by average score, with success rate (snapshot Aug 4, 2026, excluding human teleoperation): π0.5 22.9 / 12.8; InternVLA-A1 12.0 / 7.2; G0 9.0 / 4.4; Xiaomi-Robotics-0 7.9 / 3.9; X-VLA 7.6 / 3.3; GR00T-N1.7 5.9 / 1.7; π0 5.8 / 1.7; StarVLA-α 4.1 / 1.7; Spirit v1.5 1.6 / 0.6; Dexbotic-DM0 0.0 / 0.0. Each policy was evaluated over 10 trials per task, yielding 180 physical trials across 18 tasks and three embodiments.

• Real-world rankings only partially track simulation rankings. Absolute performance is low in both simulation and reality, and the paper argues that what currently binds progress is the cost of running a physical trial rather than the cost of another simulation seed.

• The clean-to-randomized gap is interpretable only because of the shared contract. Because all RoboTwin entries share one observation contract, sharp differences in the clean-vs-randomized gap across policies cannot be explained away as packing or preprocessing discrepancies.

• Reliability is treated as part of the contract. Requests carry unique identifiers with cached completed responses, so a retry after transient reconnection returns the cached result instead of re-running inference; a change in the server's instance identifier during handshake is treated as fatal to the current trial because the restarted server lost episode state.

• Not reported: the paper does not report serving-layer latency, throughput, GPU utilization, or per-policy variance/confidence intervals for the leaderboard entries; the leaderboard values are point estimates in dated snapshots.

Methodology in Plain English

The authors start from a systems observation: every (policy, environment) pair currently requires its own integration code, so cost grows multiplicatively. Their fix is to define two stable boundaries.

The first is a semantic boundary: a shared observation schema (visual inputs indexed by camera name, robot joint states, Cartesian poses in [x, y, z, q_w, q_x, q_y, q_z] convention, a language instruction, and optional metadata) and a shared action schema covering both joint-space and end-effector-space control. Robot-specific action dimensions live in embodiment configurations rather than being hard-coded into adapters. A policy adapter then implements four core operations — model construction, update_obs, get_action, and reset — plus update_obs_batch and get_action_batch for parallel evaluation. The adapter only translates between the shared interface and the model's native inference API; the network architecture, action decoder, prediction horizon, and training framework are left unspecified.

The second is an execution boundary: the policy server runs in its own software environment, the environment client stays in the benchmark's or robot's software environment, and the two talk over a WebSocket protocol with a fixed message set (HELLO, PREPARE_CASE, RESET, INFER, CALL, TRIAL_END, HEARTBEAT, CLOSE). Responsibility is split deliberately: the serving layer handles transport-level work such as deserialization and image decoding, so every adapter receives images in the same color order and layout; the adapter handles only model-specific transforms. For action-chunking policies, the environment client decides how many predicted actions to execute before requesting a new chunk, because that side knows the control rate.

Integration is then treated as conformance to a specification, validated cheapest-gate-first: static checks on configuration, imports, and scripts; then an offline closed-loop client that exercises server startup, observation serialization, action structure, batched execution, and episode reset to a deterministic finish marker without any simulator or robot; only then connection to a real environment. Because the procedure is fully specified, it is packaged as agent skills that a coding agent executes while the human configures and reviews.

To test whether this actually works, the authors ran a within-subject study: six engineers unfamiliar with XPolicyLab each connected π0.5 to RoboDojo simulation under three conditions (from scratch, manual via the standard, and with agent skills). Six participants admit exactly six orderings of three conditions, so each ordering was assigned to one participant, sessions were separated by at least one day, and participants worked only from the current condition's materials. The agent condition fixed the stack (Cursor with Opus 5, same repository checkout, skills loaded from .agents/skills/, no extra prompting scaffold). The authors note that counterbalancing cannot remove residual familiarity with π0.5 itself, which is why they read the result as an indicative case study rather than a controlled measurement of the standard in isolation.

Why This Matters

Impact on research. Reproducible comparison of robot policies has been undermined less by modeling choices than by invisible convention mismatches — camera naming, channel order, gripper scaling — silently re-fixed differently in each submission. By fixing those conventions once in a shared contract, XPolicyLab makes cross-policy differences interpretable rather than indistinguishable from preprocessing artifacts. It also complements rather than replaces existing benchmarks: those platforms define what is evaluated and under which conditions, while XPolicyLab supplies the policy-side abstraction that lets heterogeneous policies reach any of them through one interface, so no benchmark has to maintain its own model-specific serving stack.

Real-world applications (as reported):

• Bimanual manipulation benchmarking in simulation, via a RoboTwin environment client covering 50 tasks under clean and randomized settings.
• Capability-oriented simulation evaluation, via RoboDojo's 42 tasks across Generalization, Precision, Long-Horizon, Memory, and Open dimensions, run concurrently in Isaac Sim with policy servers kept separate.
• Standardized physical robot evaluation, via RoboDojo-RealEval across 18 tasks and three bimanual embodiments (ARX X5, Piper, Piper X), where the robot controller acts as the environment client and inference runs in an isolated or remote process.
• Leaderboard submission and execution workflows for official public leaderboards, with policy inference potentially running on a dedicated remote GPU server while physical control stays close to the hardware.

Industry relevance. The practical bottleneck for teams shipping robot policies is not training a checkpoint — data, dataset standardization, and training infrastructure have matured — but getting that checkpoint to run correctly in a setting where it has never run before. A reusable adapter that survives a change of simulator or robot, combined with dependency isolation that avoids reconciling conflicting software stacks, directly reduces that engineering cost. The measured drop from over five hours to two hours, and to thirty minutes with agent-executed conformance skills, is the kind of number that changes how integration work is staffed and scheduled.

Future Directions

• Whether the agent-assisted result holds beyond one case study. The study covers a single representative policy (π0.5) and one environment (RoboDojo simulation) with N=6 participants, and the authors explicitly frame it as indicative rather than a controlled measurement of the standard in isolation. Repeating it across more policies and environments would test the order-of-magnitude claim.

• Closing the simulation-to-reality ranking gap. Real-world rankings only partially track simulation ones, and absolute scores are low on both sides of the RoboDojo leaderboards. Understanding which capability dimensions predict physical performance remains open.

• Reducing the cost of physical trials. The paper argues the binding constraint is now the cost of running a physical trial rather than of another simulation seed, but explicitly notes the standard cannot remove physical variance or the scarcity of standardized robot time — only ensure scarce time goes to physical interaction rather than to rediscovering how to feed each checkpoint.

• Extending coverage and auditing at scale. With 42 policies integrated as of August 8, 2026, an open question is how the integration skill and audit skill hold up as the ecosystem grows — including whether the audit gates catch conformance failures before review as policy and environment counts increase.

Target Audience

• Robotics researchers and benchmark maintainers who need to run heterogeneous policies in their environments without building a per-policy serving stack.
• Engineers and infrastructure teams deploying VLA or other embodied foundation models onto real robots, who face conflicting software dependencies between model and robot stacks.
• Policy authors who want their checkpoint evaluated on public leaderboards with less integration overhead, and who benefit from adapters reused across RoboTwin, RoboDojo simulation, and RoboDojo-RealEval.
• Practitioners evaluating coding agents as engineering tools, since the paper reports a concrete human-supervised scaffold–implement–debug workflow with a measured effort reduction.

Authors’ abstract

Robot policy evaluation and deployment remain fragmented by model-specific software dependencies, data representations, and runtime interfaces, so that connecting N policies to M evaluation environments requires O(NM) separate integrations. We present XPolicyLab, a unified standard and open ecosystem that reduces this cost to O(N+M). XPolicyLab specifies common observation, action, and trajectory schemas together with a minimal adapter interface for observation updates, action prediction, batched execution, and episode reset, while a dependency-isolated client/server architecture separates policy inference from environment execution, so that each side retains its native software stack and may run locally or remotely. The ecosystem integrates 42 robot policies and standardizes their installation, debugging, serving, and evaluation workflows. Across these adapters, model-specific code varies by an order of magnitude while the environment-facing loop stays within a few lines of a fixed reference, confirming that the contract confines heterogeneity to the policy side. In a controlled study, conforming to the standard reduces the integration effort of a representative policy from over five hours to two hours, and packaged agent skills reduce it further to thirty minutes. The same adapters serve RoboTwin, RoboDojo simulation, and standardized real-robot evaluation through one interface. XPolicyLab is released as shared infrastructure for reproducible policy comparison and standardized deployment across simulation and physical platforms. Project website: https://xpolicylab.github.io/.