SO-101 Bench: Accessible Spatial Manipulation

AI summary

Overview

Research area: Robot learning and evaluation — specifically simulated benchmarking and sim-to-real ranking validation for low-cost robot arms, vision-language-action models (VLAs), world-action models (WAMs), and LLM-based coding agents.

Technical level: Intermediate. The paper is written clearly and its core argument (does a simulator rank policies the way a real robot does?) is easy to grasp, but following the leaderboard and ablation details assumes some familiarity with manipulation policies, fine-tuning, and correlation metrics such as Spearman rho and MMRV.

Scope (one sentence): The paper introduces SO-101 Bench, an Isaac Lab digital twin of a real SO-101 arm with four language-conditioned spatial manipulation tasks, 138 objects in three generalization splits, and 916 scored episodes, and uses it to show that simulated policy rankings match rankings on the physical arm.

What This Paper Is About

Low-cost SO-101 arms are now the most common entry point to robot learning, yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box, and the existing SO-101 benchmarks run only on physical hardware with simple tasks and few held-out objects. The paper builds a simulated benchmark — a digital twin of a real SO-101 setup — so that policies can be evaluated cheaply and repeatably, and then tests whether that twin actually predicts what happens on the real robot. The goal is a benchmark whose rankings carry over to hardware while still measuring the skills that aggregate success hides: spatial placement, novel objects, rotate-to-fit planning and memory for occluded objects.

Key Contributions

1. SO-101 Bench, a digital-twin benchmark for language-conditioned spatial manipulation on a low-cost arm, with instance- and class-level object splits, ROBUST, PLAN and MEM capability subsets, and automatic scoring from simulator state. It is released with the so101_bench_sim (3,629 simulated episodes) and so101_bench_real (approximately 4,400 real episodes) datasets.

2. Evidence that the twin ranks policies as the real robot does: condition- and failure-level correspondence between independently trained real-only and sim-only policies, plus an exact sim-to-real ranking of five policies (Spearman rho = 1.00, MMRV = 0.000) run with identical weights and no real data from the authors' setup — these policies work out of the box in an unseen real environment with unseen objects.

3. A leaderboard of nine VLAs and WAMs and two LLM agents, where every learned policy is fine-tuned on the same 3,629 simulated demonstrations under an equal 36 h x 4 x A100 compute budget, showing that spatial placement, unseen object classes and rotate-to-fit planning remain open.

4. Harness and model ablations showing that a code-as-policy harness that hands grasping to a learned policy outperforms either the harness or the policy alone, that direct action generation by the agent is expensive and ineffective, and that the choice of language model also matters.

Main Findings

• The best learned policies reach about 61%. InternW0-Δ (61.4, 95% CI [58.5, 64.1]) and Galaxea 0.5 (60.8, [57.9, 63.6]) are statistically tied at the top, followed by GR00T N1.7 (45.8), ImageWAM (43.0), pi-0.5 (41.0) and LaWAM (39.9). Both MolmoAct2 variants score below 25 and FLUX 3 Action scores 8.0.

• Grasping familiar objects is largely solved; spatial tasks are not. The strongest learned policies bin a single object in 90–93% of episodes and place 90% of objects in 4-object bins, but next-to and between success is only 34–44% for the same policies.

• Object novelty compounds the difficulty. On spatial tasks, InternW0-Δ falls from 72% on seen objects to 18% on unseen classes for next-to, and from 56% to 19% for between; Galaxea 0.5 falls from 71% to 19% and from 63% to 12%. Pooled over tasks, the six strongest learned policies lose 30–43 points between seen objects and unseen classes.

• PLAN and MEM isolate distinct weaknesses. No learned policy exceeds 17% when a grasped object must be rotated to fit (GR00T N1.7 is best), while the LLM agent reaches 75%. On MEM, learned policies reach at most 47% (InternW0-Δ).

• ROBUST separates policies by visual robustness rather than skill. Galaxea 0.5 and GR00T N1.7 keep their overall success under held-out backgrounds and robot colours, whereas LaWAM drops from 39% to 18% and ImageWAM from 43% to 34%.

• LLM agents lead the leaderboard. GPT-6.1 Sol, writing code in the PyRUA-Lean harness and handing grasps to Galaxea 0.5, scores 83.2 [80.7, 85.7]; DeepSeek V4.1 Flash-1017 scores 66.8 [63.7, 69.9]. Sol's success varies by only five points across object splits (85/82/80%).

• The harness matters more than the model. On 60 paired episodes, Sol succeeds on 35% (21/60) with direct tool calls, 67% (40/60) with PyRUA-Lean and 85% (51/60) with the Galaxea 0.5 hand-off, against 53% (32/60) for Galaxea 0.5 alone. The hand-off is also cheaper, with API cost falling from $18.44 to $14.68 over the 60 episodes.

• The model matters too. Holding the PyRUA-Lean + Galaxea 0.5 harness fixed: GPT-6.1 Sol 85% (51/60), Gemini 4 Argon 77% (46/60), DeepSeek V4.1 Flash-1017 65% (39/60), MiMo v2.6 Pro 55% (33/60) — all above Galaxea 0.5 alone at 53%.

• The twin reproduces the difficulty structure of the real benchmark. An independently trained sim-only GR00T N1.6-3B policy and a real-only one order the 15 task x split conditions similarly (Pearson r = 0.86 [0.71, 0.91], Spearman rho = 0.74 [0.52, 0.91]) and shift from placement to semantic and grasping failures in the same way as object novelty increases.

• Sim-to-real transfer holds the ranking exactly. With identical weights, five policies rank identically in simulation and on the physical arm: GPT-6.1 Sol (PyRUA) + Galaxea 0.5 84% (38/45) vs. 64% (29/45), Galaxea 0.5 67% (30/45) vs. 53% (24/45), GR00T N1.7 56% (25/45) vs. 24% (11/45), pi-0.5 49% (22/45) vs. 18% (8/45), and MolmoAct2-SO101 24% (11/45) vs. 16% (7/45). Spearman rho = 1.00, MMRV = 0.000 (95% CI [0.000, 0.169]), Pearson r = 0.90 [0.66, 0.97].

• Simulation is optimistic, by different amounts per policy. Every policy succeeds more often in simulation, by 9 to 31 percentage points (20 for the agent). Almost all paired disagreements are episodes that succeed in simulation and fail on the real robot (57, against 10 the other way). Absolute simulated success rates should be read as an upper bound; comparisons between policies transfer.

• The agent's lead transfers to hardware. On the real robot the agent succeeds on 29 of 45 episodes against 24 for Galaxea 0.5 (exact McNemar p = 0.30; in simulation 10 against 2, p = 0.04). It solves 8 of 9 real next-to episodes (Galaxea 0.5: 2) and 7 of 15 unseen-class episodes (Galaxea 0.5: 4), and eight real episodes were solved by the agent and none of the four learned policies. None of its 16 real failures involved the wrong object.

• Training scale appears to drive leaderboard position. Under the equal budget the number of fine-tuning samples each policy sees varies by a factor of 30. The strongest learned policies see at least a full epoch (InternW0-Δ 1.2, Galaxea 0.5 1.6), whereas MolmoAct2 (0.5–0.6 epochs) and FLUX 3 Action (0.2) appear under-trained; these rankings are at a fixed compute budget and should not be read as capability ceilings.

• Prior exposure to the embodiment did not help. Starting MolmoAct2 from its SO-100/101 post-trained checkpoint did not beat the base checkpoint (20.9 vs. 23.8, overlapping intervals) under this protocol.

Methodology in Plain English

The authors built a software replica ("digital twin") of a real SO-101 tabletop setup in the Isaac Lab simulator, scanning the room and reconstructing household objects from photographs. Because the real arm's joint calibration is aligned to the simulated kinematics, the same policy weights can run in both places.

They defined four household-style tasks, given as natural-language instructions: put every object in the bin; put one object next to another; put one object between two others; and move an object in a direction until it reaches the object in its way. These are scored as five task families, and all success and failure conditions are checked automatically from the simulator state using distances on the table, contact forces and time limits, so no human judgement is needed.

To test generalization, the 138 objects are split three ways: objects seen in training, new instances of seen classes, and entirely new classes (37 of which are kept private to prevent test-set leakage). Three overlapping subsets probe held-out appearances, rotate-to-fit planning and memory for occluded objects.

Every learned policy is fine-tuned on the same 3,629 teleoperated simulated demonstrations under the same compute budget, with checkpoints chosen on a held-out validation set, so the leaderboard compares policies on equal footing rather than on each one's own recipe.

The validity argument comes in two steps. First, the authors train two GR00T N1.6-3B policies independently — one on real demonstrations evaluated on the physical arm, one on simulated demonstrations evaluated in the twin — and check whether they find the same conditions hard and fail in the same ways. Second, and more strictly, they take five policies that never saw real data from this setup, freeze the weights, and run them both in the twin and on the physical arm in an unseen room with unseen objects, using layouts matched by projecting simulated object footprints into the real overhead camera view.

Finally, they compare LLM agents against learned policies on the same episodes, deliberately varying one thing at a time — the harness (direct tool calls, code-as-policy, or code plus a learned grasp policy) and the language model — on a fixed set of 60 paired episodes selected before any agent was run.

Why This Matters

Impact on research. Real-robot evaluation is expensive, noisy and build-dependent — servo offsets differ between arms, and real-robot studies typically afford only tens of rollouts per condition. A validated twin makes it possible to run 916 scored episodes with automatic scoring, held-out object classes and confidence intervals at a fraction of the cost. The paper also supplies something the sim-to-real literature has largely lacked: a test in which sim-trained weights, not real-trained ones, cross the domain boundary and keep their ranking, and a direct measurement of how much simulation overestimates success.

Real-world applications:

• Choosing a policy for a low-cost robot deployment — the leaderboard tells a practitioner roughly what to expect from nine off-the-shelf VLAs and WAMs, and which skills (spatial placement, novel objects, rotate-to-fit) they will need to solve themselves.
• Warehouse and lab automation with pick-and-place arms — the bin, next-to and between tasks map directly onto kitting, sorting and shelf-placement work where same-class distractors and unfamiliar objects are routine.
• Assistive and service robotics — the MEM subset (remembering an occluded object's position and 3D footprint) and the directional-move task reflect the kind of interrupted, partially occluded manipulation such systems face.
• Cheap, repeatable regression testing during model development — teams can iterate on checkpoints in simulation and trust that the ordering of candidate policies will hold on hardware.

Industry relevance. The paper's central claim — that an LLM agent writing code and delegating grasping to a learned policy beats either component alone (85% vs. 53% for the policy alone on the same 60 episodes) — is a concrete architectural recommendation, and it comes with a cost table ($14.68 for GPT-6.1 Sol versus $74.25 for Gemini 4 Argon on the same episodes). The caveat that simulated agent evaluations pause the simulator and therefore hide real-time cost is equally relevant to anyone planning a deployment: median elapsed wall time was 197 s for the agent versus 38 s for Galaxea 0.5 over the same 45 layouts.

Future Directions

• Closing the open skill gaps. Spatial placement (next-to and between at 34–44% for the strongest learned policies), unseen object classes (30–43 point drops from seen to unseen class), rotate-to-fit planning (no learned policy above 17%) and occlusion memory (at most 47%) are all explicitly left open.

• Making the twin quantitatively, not just ordinally, faithful. Simulation overestimates success by 9 to 31 percentage points, and the authors trace part of this to physical mismatches such as the far lesser mass of the real plastic bin and real objects scaled to 0.7–0.9 times their size. Correcting those mismatches would let simulated scores be read as predictions rather than upper bounds.

• Strengthening the ranking evidence. The sim-to-real study uses five policies and 45 episodes, so the confidence interval on MMRV remains wide ([0.000, 0.169]); the authors treat the result as evidence of correct ranking rather than precise calibration, and extending it to more policies and more real rollouts is a natural next step.

• Separating capability from compute budget. The three weakest learned policies saw well under one epoch of fine-tuning, and MolmoAct2's published recipe uses about five times as many samples while FLUX 3 Action's LoRA run used less than a tenth of its released full fine-tuning recipe — so re-running them at their native scale would test whether the leaderboard ordering is a property of the models or of the protocol. The real-time cost of LLM agents, hidden by pausing the simulator, is a further open question for practical deployment.

Target Audience

Robotics and embodied-AI researchers who need a cheap, reproducible evaluation harness for manipulation policies; engineers working with low-cost arms such as the SO-101 who want to

Authors’ abstract

Low-cost arms such as the SO-101 have become the most common entry point to robot learning, yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box, and existing SO-101 benchmarks run only on physical hardware with simple tasks and few held-out objects. We introduce SO-101 Bench, a simulated benchmark for language-conditioned spatial manipulation built on an Isaac Lab digital twin of a real SO-101 setup. It has four tasks scored as five families, 138 objects in seen, unseen-instance and unseen-class splits, 916 evaluation episodes with subsets for held-out appearances, rotate-to-fit planning and occlusion memory, and 3,629 simulated demonstrations on which every learned policy is fine-tuned under an equal compute budget. The twin ranks policies as the real robot does: five policies trained without real data, four VLAs and an LLM agent, keep their exact ranking when the same weights run on the physical arm in an unseen environment with unseen objects (Spearman rho = 1.00, MMRV = 0.000), although simulation overestimates success. Across nine VLAs and world-action models, the best learned policies score near 61%, with spatial placement, unseen object classes and rotate-to-fit planning remaining open. An LLM agent that writes code and hands grasping to a learned policy scores 83.2%, and an ablation shows that this lead comes from the choice of harness and the choice of language model.