ArmnetBench v0.1: Parallel Real-World Evaluation of Manipulation Policies on a Low-Cost Arm Farm

AI summary

Overview

• Research area: Robot learning / robot manipulation policy evaluation (physical real-world benchmarking infrastructure for manipulation policies).
• Technical level: Intermediate. The paper is written to be readable, but assumes familiarity with imitation-learning policies, vision-language-action models, and benchmark design terminology.
• Scope: ArmnetBench v0.1 is a real-world robot manipulation benchmark run on a managed fleet of low-cost SO-101 cells, comparing 7 policies across 12 tasks under a shared 50-demonstration training budget per task, and releasing 3,118 labelled core episodes (plus 600 additional rollouts).

What This Paper Is About

Evaluating generalist robot manipulation policies in the real world is slow and expensive: every rollout needs physical hardware and an operator to set up, reset, and score it, which typically limits studies to a few dozen trials on one robot in one lab. ArmnetBench v0.1 addresses this by running evaluations in parallel on a fleet of low-cost SO-101 cells under light on-site supervision, so that many tasks, policies, and both single-arm and bimanual configurations can be compared under one protocol.

The goal of v0.1 is not to establish each method's capability ceiling, but to validate the arm-farm evaluation substrate end to end and produce an initial leaderboard plus a released, quality-labelled corpus of rollouts.

Key Contributions

1. The Armnet arm farm: a managed fleet of low-cost single-arm and bimanual SO-101 cells that evaluates policies in parallel under light on-site supervision.
2. ArmnetBench v0.1: a 12-task suite (8 single-arm, 4 bimanual) and a shared-budget evaluation protocol for comparing 7 policies, using three-way operator scoring.
3. A released labelled corpus: 3,718 labelled episodes in total, comprising 3,118 core benchmark episodes in LeRobot v3.0 and RoboMeter formats plus 600 additional rollouts.
4. Released checkpoints: the exact policy checkpoint evaluated is released for every benchmark task–policy pair, on the HuggingFace Hub.

Main Findings

• Leaderboard ordering: π0.5 leads both embodiments with 47.6% strict success overall (359 scored rollouts) and 51.5% success+suboptimal. π0 follows at 35.1% / 40.4% (359 rollouts), then GR00T N1.7 at 29.4% / 33.1% (360), Diffusion Policy at 26.7% / 29.7% (360), ACT at 19.2% / 21.1% (360), MolmoAct 2 at 18.9% / 21.7% (360), and SmolVLA at 15.0% / 19.2% (360).
• Embodiment-dependent standings: π0.5 leads single-arm at 45.4% and bimanual at 52.1%. On bimanual tasks the middle of the ranking reshuffles: Diffusion Policy rises to second at 35.8%, while ACT falls to 2.5%. The paper states relative standings should not be extrapolated from single-arm results.
• One task defeats every policy: no policy succeeds on the contact-rich cable_clip task, whereas the best task–policy pairs reach 60–86%.
• Labels are nearly binary in practice: of 2,518 policy rollouts, 89 (3.5%) carry the suboptimal label, meaning operators used the graded category conservatively.
• Corpus composition: 3,118 labelled core episodes = 600 human-teleoperated reference episodes (12 × 50) plus 2,518 policy rollouts (7 × 12 × 30, minus 2 dropped). Labels comprise 1,290 successful, 1,739 failure, and 89 suboptimal. Every episode has 3 synchronised camera views at 20 fps.
• Deployment scale: v0.1 ran on 3 co-located cells (2 single-arm: cell-1, cell-3; 1 bimanual: cell-8), collecting 2,520 benchmark rollouts total (840 per cell) with 2 removed for manual reset errors. Three cells were supervised concurrently per operator, with an estimated active operator time of about 10 s per rollout (reset plus score), described as a retrospective estimate.
• Cost: per-cell totals are approximately $359 (single-arm) and $477 (bimanual), amortising the $259 SO-ARM101 kit.
• Reproducibility is uneven: execution reproducibility is strong (released container images and checkpoints reproduce the policy stack); within-cell scene consistency is moderate (rigid mounts and recorded pan/tilt settings fix camera-to-arm geometry, but ambient lighting varies); cross-cell physical reproducibility remains unanswered because each task ran on one cell.

Methodology in Plain English

The authors built an arm farm from low-cost SO-101 robot arms. Each cell holds one or two 5-DoF follower arms, three cameras, and a Raspberry Pi 5 edge device in a fixed workspace bounded by A3 cardboard sheets (297 × 420 mm) — three sheets for single-arm cells, four for the wider bimanual cell. Single-arm cells use front, top, and wrist cameras; bimanual cells use top, left_wrist, and right_wrist. Observations are camera views plus five arm-joint positions and one gripper position per arm; actions are target arm-joint and gripper positions. A networked Shelly plug allows remote power scheduling and recovery.

A user submits a policy image, embodiment, and task. The workstation builds the image and runs it in an isolated container, keeping policy dependencies separate from cell software, and streams commands over the local network to the edge Pi driving a matching arm. A cloud backend schedules jobs and streams logs and video to a fleet-management control panel, where an operator monitors the cells, resets scenes, and scores each rollout. Between resets, cells execute autonomously.

The benchmark contains 12 tabletop manipulation tasks spanning stacking, insertion, deformable objects, pick-and-place, and articulated objects. Every task has 50 human-teleoperated reference demonstrations used to train or fine-tune each policy. Seven policies are compared: two specialist imitation policies (ACT and Diffusion Policy) and five vision-language-action models (SmolVLA, π0, π0.5, GR00T N1.7, and MolmoAct 2). Each task–policy pair targets 30 rollouts. Before each rollout the operator resets the scene and randomises object placement within a task-specific range; initial states are sampled independently rather than matched across policies. Operators assign one of three labels: successful, suboptimal, or failure. The 50 demonstrations per task are successful by construction.

Data is released in two formats. LeRobot v3.0 stores synchronised state, action, and packed AV1 video with sparse terminal reward and done fields (next.reward, next.done) and episode metadata (success, success_class, policy_type, policy_repo_id). The RoboMeter export stores one video per camera, the quality_label, and an embedding of the task instruction; including the auxiliary rollouts it contains 3,718 episodes (roughly 26 hours). The quality labels are intended to support reward modelling, quality-conditioned policy training, and action-conditioned world models.

Why This Matters

Impact on research. Real-world evaluation is described as a bottleneck in developing generalist manipulation policies. By reducing per-trial cost through parallel low-cost cells, this work enables broader coverage across tasks, policies, and embodiments under one protocol, with every scored rollout released as data. The paper also notes that unreported choices such as camera placement, lighting, object initialisation, checkpoint selection, and task-specific tuning can inflate headline success rates and hinder independent reproduction. Benchmarking prior work mostly provides blueprints for independently rebuilt setups; ArmnetBench is characterised as a managed fleet with graded three-way scoring, a human in the loop, and a labelled evaluation corpus.

Potential applications enabled by the design (as implied by the paper's infrastructure and releases):

• Comparing manipulation policies under a common data budget on physical hardware rather than in simulation.
• Reusing quality-labelled trajectories to train reward models or predictive world models.
• Training or fine-tuning policies on mixed-quality data, including suboptimal successes.
• Serving as a template for low-cost, multi-cell evaluation setups using widely available open hardware such as SO-101 and LeRobot tooling.

Industry relevance. The benchmark targets the cost of physical evaluation, which is a practical barrier for teams developing manipulation policies. It packages hardware costs (approximately $359 per single-arm cell, $477 per bimanual cell), a containerised policy-execution pipeline, a cloud backend for scheduling and monitoring, and open data releases in two formats, all of which map onto engineering workflows for robot learning teams. The measured operator workload (about 10 s of active time per rollout, three cells supervised concurrently per operator) is the metric the authors identify as the primary target for future reduction.

Future Directions

• Reduce operator time per rollout — the stated primary future objective. Automated scene reset would remove the main manual step and make initial states repeatable.
• Log initial object positions to verify each reset, since v0.1 randomised placement within task-specific ranges but did not record positions, preventing analysis of spatial success patterns.
• Add a reward-model labeller to reduce manual scoring and allow one operator to supervise more cells.
• Secondary goals: a broader task suite, per-cell lightboxes, camera-framing checks, and a shared task–policy slice run across cells to measure cross-cell reproducibility — the level of reproducibility the paper says remains unanswered because each task ran on one cell.

Other open issues raised by the limitations: standardising per-task wall-clock limits (v0.1 relied on operator judgement for rollout termination), managing object deterioration over long evaluations (for example the eye-drops carton and cable-task velcro), and accounting for physical cell changes (on cell-3 the front camera was misaligned for every policy except MolmoAct 2, and the bimanual right_wrist view was slightly blurry for every policy except MolmoAct 2).

Target Audience

Researchers and engineers working on robot manipulation policies, imitation learning, and vision-language-action models who need physical evaluation rather than simulation. It is also relevant to benchmark designers and reproducibility-focused groups, to teams building low-cost robot hardware or fleet-management tooling, and to readers interested in datasets of quality-labelled robot trajectories for reward modelling and world-model training. Readers wanting a purely algorithmic treatment of policy learning will find little here: the paper is about the evaluation substrate, protocol, and released corpus.

Authors’ abstract

Real-world evaluation is a bottleneck in developing generalist robot manipulation policies. Each rollout requires physical hardware and an operator to set up, reset, and score it. We introduce ArmnetBench v0.1, a benchmark run on a fleet of low-cost SO-101 cells under light on-site supervision. v0.1 validates this arm farm end to end and compares 7 policies across 12 tasks with both single-arm and bimanual configurations. Each policy is trained or fine-tuned on 50 demonstrations per task; the benchmark contains 2,518 policy rollouts and 600 reference demonstrations. All 3,118 episodes carry a three-way label (successful, suboptimal, or failure). Policy rollouts are human-scored, while demonstrations are successful by construction. Beyond evaluation, its quality-labelled trajectories support downstream learning, from reward and predictive world models to policies trained on mixed-quality data. The leaderboard is an initial comparison under this shared budget. We release the 3,118 core episodes in LeRobot v3.0 and RoboMeter formats.