RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild

AI summary

Overview

Research area: Robot learning and embodied AI — specifically, how to benchmark general-purpose vision-language-action (VLA) robot policies in real physical environments rather than in simulation.

Technical level: Intermediate to Advanced. The core ideas (distributed physical trials, pairwise comparisons, fine-tuning on demonstrations) are accessible, but the ranking methodology relies on item-response theory (a Rasch model) and Bradley–Terry–Davidson models that require some statistical background.

Scope (one sentence): RoboColosseum is a distributed, multi-embodiment benchmark that evaluates five VLA policies and three frontier AI agents across four robot platforms, 20 fine-tuning tasks, and 5,699 blinded pairwise trials conducted at 124 real-world sites.

What This Paper Is About

Robot policies that are meant to be general-purpose must work across different robots, different tasks, and different environments — but the very variation that makes physical trials informative also makes them hard to reproduce, and real-world evaluation is expensive, labor-intensive, and usually run at small scale on a single robot in a single lab. The authors build RoboColosseum, a network of locally operated robots at many sites that all follow shared hardware specs, task definitions, and evaluation protocols, so that policy comparisons can be made at scale in messy, uncontrolled settings while still being fair. The goal is to answer where today's generalist robot models actually generalize, how well they adapt to new tasks from a fixed demonstration budget, and how far they get on tasks they ultimately fail.

Key Contributions

1. A distributed physical evaluation framework with two complementary tracks. The Open Track measures zero-shot capability on in-the-wild tasks chosen by each site, using blinded pairwise comparisons under matched local conditions, with no task-specific adaptation. The Fine-tune Track measures adaptation on 20 predefined tasks (five per embodiment), where every policy receives the same 100 demonstrations per task.

2. Operational infrastructure and protocols, not just a task list. The paper specifies standardized observation and action spaces per embodiment (SO-101, Franka in the DROID configuration, Bimanual-YAM, Unitree G1), teleoperation interfaces, a policy-serving layer with local (RTX 5090 laptop) or remote inference, up to 30 full-time robot operators, privacy processing, and a pledge of 10,000 hours of free evaluation to academic labs.

3. A difficulty-aware ranking method for in-the-wild trials. The Difficulty-Calibrated Rasch (DCR) model jointly estimates each policy's latent ability and each task's latent difficulty from physical success outcomes, using human difficulty ratings (1–10) as a prior on task difficulty, and reports ability on an Elo-style scale centered at 1500. It is compared against Bradley–Terry–Davidson, sequential Elo, and average pairwise score baselines.

4. A large empirical study spanning 41 days (17 September 2026 to 28 October 2026). 5,699 Open Track comparisons at 124 sites across offices, libraries, parks, dormitories, homes, food courts, and factories in Singapore, Malaysia, and China, plus full Fine-tune Track results on 20 tasks with 25 trials per policy–task pair.

Main Findings

• Frontier AI agents rival specialized VLA policies, but inconsistently. Across 901 paired agent-versus-VLA evaluations, agents achieved greater task progress in 394 comparisons, VLAs in 265, and 242 were tied — an average agent advantage of 5.3 percentage points. That advantage splits by agent: Claude Opus 5.5 gains 12.1 points and GPT-6 Astra 11.0 points, while Grok 4.7 trails its VLA counterparts by 7.5 points. On DCR ratings, GPT-6 Astra has the highest point estimate on DROID/Franka, while Claude Opus 5.5 leads on SO-101 and Bimanual-YAM.

• Nobody reliably completes open-world tasks. In the Open Track, MolmoAct2 has the highest expected comparison score among the five VLA policies in all six manipulation categories (Pick/Place, Stack/Arrange, Open/Close, Reorient, Deformable, Tool/Interaction), ranging from 58% to 77% — yet its own success rates are only 6% to 13%, and no evaluated system exceeds 13% success in any category.

• Progress collapses with task difficulty. Full-task success falls from 30.7% on tasks rated 1–4 to 0.4% on tasks rated 9–10, and mean normalized progress on unsuccessful executions falls from 56.6% to 3.5% over the same range. Failures happen well before the final success condition, not just at the last step.

• Fine-tuning winners depend on the embodiment. On SO-101, GR00T N1.7 leads with an average progress score of 89.0 and success rate of 82.4. On DROID/Franka, π0.5 leads with 91.2 progress and 84.0 success. π0.5 also leads on Bimanual-YAM, reaching 96–100% success on four of the five tasks and 72% on stack_cups.

• Adaptability is not the same as broad adaptability. Individual tasks produce dramatic reversals: on DROID/Franka, LingBot-VLA v2 gets 100% success on open_drawer but 0% on marker_in_holder, while MolmoAct2 shows the opposite pattern at 0% and 88%. G0.5 reaches 100% on both drawer and microwave on Bimanual-YAM but only 20% on stack_cups.

• Rankings do not transfer across embodiments. Policy rankings show weak agreement across embodiment-specific task suites, with Kendall's W = 0.24.

• Open Track performance poorly predicts fine-tuned performance. There are substantial rank reversals between the two tracks on all three embodiments shared between them.

• The humanoid is far from solved. On the Unitree G1, no policy completes any of the five tasks, yet average progress still ranges from 13.3% to 23.2% across policies — differences that a success-only metric would erase entirely.

• Rankings stabilize with scale, and active matchmaking saves trials. Open Track rankings stabilize as trials accumulate, and active matchmaking reaches roughly 90% agreement with the full-data ranking after 400–500 evaluations, compared with roughly 1,000 under uniform sampling.

• Safety stops are part of the data. Two evaluations were halted for safety: LingBot-VLA v2 on the YAM drawer after 1 trial, and GR00T N1.7 on the YAM microwave after 9 trials; unexecuted trials are scored zero.

Methodology in Plain English

The authors did not build one big lab. They built a network: many robots in many places, all set up to the same specification, all talking to shared policy-serving software. Each robot type — a cheap SO-101 arm, a Franka arm in the DROID configuration, a two-armed YAM, and a walking G1 humanoid — has a fixed recipe for what the cameras see and what the actions mean, so the same policy can run at any site with that robot.

They then ran two kinds of study. In the Fine-tune Track, they wrote task cards for 20 tasks (five per robot) covering pick-and-place, stacking, reorienting objects, drawers and laptop lids, towels and clothing, pouring, sweeping with a dustpan, bimanual coordination, and loco-manipulation. For each task they collected 100 teleoperated demonstrations at the evaluation site using that same robot and environment, released them publicly, and gave every policy exactly the same 100 demonstrations. Each policy then followed its own official fine-tuning recipe until convergence — the data was held fixed, the training procedure was not. Every policy ran 25 trials per task, scored on both success rate and a progress score: the fraction of that task's progress criteria satisfied, with each criterion weighted equally, so partial credit survives even when the task is failed.

In the Open Track, a human evaluator at a site invents a task using whatever is around, writes a natural-language instruction, and the system secretly assigns two eligible models labeled A and B. Both run back-to-back on the same robot, same objects, same setup, same timeout. The evaluator, who never learns which model is which, reports a preference (A wins, B wins, or tie), progress scores, free-text feedback, and a perceived difficulty from 1 to 10.

To turn those pairwise outcomes into a leaderboard, the authors use a Rasch-style model where each policy has an ability and each task has a difficulty, and the chance of success is a logistic function of the difference. Human difficulty ratings act as a prior on task difficulty rather than a hard weight, so physical outcomes can move a task's inferred difficulty away from its rated category. Because robot trials are expensive, they also use active matchmaking — borrowed from Chatbot Arena — that prioritizes model pairs with the largest expected reduction in uncertainty, instead of sampling pairs uniformly. Reliability is checked with Spearman's correlation, pairwise ordering agreement, and repeated subsampling at increasing evaluation budgets.

Why This Matters

The paper argues that robot learning has no equivalent of the human-preference benchmarks that drove progress in language and vision, and that real-world robot studies typically rely on relatively few rollouts, making conclusions sensitive to trial selection. RoboColosseum shows a practical path to statistically reliable physical evaluation at scale — and shows that success rate alone hides real differences, since the G1 humanoid results separate policies only through partial progress.

Real-world applications that depend on these findings:

• Manufacturing and warehouse deployment, where the paper already runs trials in factories in China and where knowing a policy's success rate on high- versus low-difficulty tasks predicts where automation is safe to deploy.
• Home and service robotics, since evaluations took place in homes in Malaysia, dormitories, food courts, libraries, and parks — the actual messiness of consumer environments rather than a lab bench.
• Robot foundation model development, where a shared, blinded, difficulty-calibrated leaderboard gives model builders a target that is not tied to one embodiment or one lab's camera setup.
• Academic accessibility, through the pledged 10,000 hours of free evaluation, letting labs without hardware test policies on real robots across multiple embodiments.

Industry relevance: the result that API-accessible frontier agents match specialized VLA policies on physical manipulation is directly relevant to companies deciding whether to train a bespoke policy or wrap an existing frontier model. The rank reversals between the Open and Fine-tune Tracks matter for procurement: a model that wins a zero-shot leaderboard may not be the one that adapts best to your task after fine-tuning.

Future Directions

• Diagnosing partial progress and failure modes. Since policies fail long before the final success condition on difficult tasks, the natural next step is finer-grained sub-step analysis of where execution breaks down — the paper's progress criteria already provide the scaffolding for this.

• Closing the humanoid gap. No policy completed any of the five G1 tasks despite 13.3%–23.2% average progress. Extending the task registry and scaling evaluations on legged, loco-manipulation embodiments is an open problem.

• Making rankings portable across embodiments and tracks. With Kendall's W = 0.24 across embodiment suites and rank reversals between tracks, understanding what a leaderboard position actually predicts — and designing protocols that surface adaptability rather than only zero-shot skill — remains unresolved.

• Scaling the network and the sample-efficiency work. The framework is described as a growing fleet of over 100 robots across the United States, United Kingdom, Singapore, Malaysia, and China, with the current study covering 124 sites. Extending coverage and pushing active matchmaking further below the 400–500 evaluation mark needed for roughly 90% ranking agreement are direct extensions.

Target Audience

Robot learning researchers and benchmark designers who need to evaluate generalist policies outside simulation; engineers at robotics companies comparing foundation models for a specific embodiment; evaluation and statistics researchers interested in preference-based ranking under costly, noisy physical trials; and research labs without dedicated hardware, for whom the shared infrastructure and pledged free evaluation hours lower the barrier to running real-world experiments. Readers who want a concrete picture of what today's VLA models and frontier agents can and cannot do in unstructured environments will also find the empirical results directly useful.

Authors’ abstract

General-purpose robot policies must work across robots, tasks, and environments, yet the variation that makes physical trials informative also makes them difficult to reproduce. We introduce RoboColosseum, a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments. Our initial study evaluates five vision-language-action (VLA) policies and three frontier AI agents across four embodiments (SO-101, Franka, Bimanual-YAM, and G1 Humanoid). The Open Track compares models without task-specific adaptation in blinded, pairwise trials under matched local conditions. The Fine-tune Track measures how well VLA policies adapt to 20 tasks across all four embodiments when each policy receives the same 100 demonstrations per task. Across 5,699 pairwise Open Track comparisons at 124 sites, frontier agents rival specialized VLA policies, while all systems struggle to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM. Policy rankings show weak agreement across embodiment-specific task suites (Kendall W = 0.24). Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. Measuring partial progress reveals differences that task success alone misses. On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies. Open Track rankings stabilize as trials accumulate, and active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. These results show how distributed physical evaluation can reveal where robot policies generalize, how well they adapt to new tasks, and how far they progress on tasks they fail to complete.