Embodied First Exam

AI summary

Overview

Research area: Embodied AI and robotics benchmarking — specifically, evaluating general-purpose LLM-driven coding agents as robot controllers.

Technical level: Intermediate. Readers should be comfortable with robot learning terminology (visuomotor policies, simulation, task success rates), but the paper's framing is evaluation-centric rather than method-heavy.

Scope (one sentence): The paper introduces the Embodied First Exam (EFE), a unified simulation benchmark of 1,156 tasks across 35 suites and 47 robots that measures how well general-purpose coding agents can control robots under realistic physical-world constraints.

What This Paper Is About

Existing robotics benchmarks were built for a different paradigm: training visuomotor policies on a fixed observation-to-action interface. Coding agents (LLMs with shells, file systems and the ability to write and run code) suggest a third route to robot control — they can act directly, write programs that act, or train a policy — but no benchmark measures whether they can do so reliably when they only see what a real robot's sensors show and cannot undo their mistakes. EFE asks what robotic tasks general-purpose coding agents can solve, under what physical-world constraints, and where they fail.

Key Contributions

1. A large, agent-native exam. 1,156 tasks in 35 suites on 47 robots of nine types, combining 26 suites (745 tasks) adapted from existing robotics benchmarks with 9 new suites (411 tasks) built for the exam. The new suites go beyond pick-and-place into contact-rich deformable manipulation (RoboARC, DexHandExam, RoboPaint), dynamic and competitive tasks (RoboDynamicExam, two Embodied Olympics suites), and abilities like image-only goals (iSpatialGym), safety screening (Embodied Safety Exam) and adapting controllers to hidden dynamics (RoboEngineer).

2. A three-setting evaluation protocol. Each task can be run in a Standard setting (only sensor observations; simulation paused while the agent reasons; no resets unless a budget is granted), a Realtime variant (the world keeps running while the agent decides), and a Privileged setting (full simulator state and unlimited resets, used as an approximate upper bound and for generating training data). All settings of a task share the same initial scene, success rule and time limit, so results are paired.

3. Unified infrastructure that leaves the control approach to the agent. Built on Harbor, the container-based evaluation framework of Terminal-Bench, each task runs in a container with its own workspace; a verifier outside the agent's control grades every attempt. Given a natural-language instruction, the agent chooses among direct control, code to control, or training a policy.

4. Stage-level failure analysis. 427 of the 456 core tasks are decomposed into task graphs of stages (e.g., fridge opened → tomato inside → fridge closed). Failure stages and failure phenomena are labeled by separate coding agents, enabling comparisons across agents, settings and stage categories.

Main Findings

• A large Privileged-to-Standard gap. On the 391 hard core tasks that have both settings, the strongest agent (GPT-6 Astra in Codex) solves 77% of Privileged tests but 30% of Standard tests. On the 456 core tasks it reaches 34% in the Standard Setting. The three strongest agents — GPT-6 Astra, Claude Opus 5.5 and Gemini 4 Argon — solve 24.1% to 33.6% of the 456 Standard tests and 48% of them jointly.

• Realtime is harder than Standard. On the 54 tasks with both tests, GPT-6 Astra solves 23 Standard and 17 Realtime tests, Claude Opus 5.5 solves 10 and 4, and Gemini 4 Argon solves 15 and 5. 63% of these Realtime tests remain unsolved by any agent, versus 50% of the Standard tests on the same tasks.

• Resets matter more than exact state. In controlled runs, adding an exact-object-state query to the Standard interface lifts GPT-6 Astra from 28% to 45%; adding unlimited resets lifts it to 53%; both together reach 68%, against 77% in Privileged. With resets, 66% of successes come from episodes started at a saved state.

• Precision stages collapse without privileged access. Failure rates rise by 24 to 46 percentage points at precise stage types and by 1 to 20 points at coarse types. Assembling rises from 34% to 80% and placing from 23% to 35%. At precise stages, resets help (36% of 91 failures solved) but exact state alone does not (21%, versus 19% for a run with no added feature); at coarse stages state alone solves 34% of 97 failures versus 14%.

• Failures cluster at the same stage. For the three strongest agents, 56% of failure pairs on a test share a stage (31% by chance); across all eight agents the figure is 61% versus 37%. The most common failure stage is the first stage in 65% of the 264 Standard tests that have a single most common stage.

• Failure stage is largely setting-invariant. Of 83 tasks (after excluding 105 where the common stage is first in both settings), 47 (57%) have the same most common failure stage in Privileged and Standard, versus 9% chance.

• Some capabilities fail hardest. Once reached, material-changing stages (e.g., wiping) fail at 66%, body-moving stages (e.g., walking) at 45%, and mechanism stages (e.g., opening a door) at 20%.

• Weaker agents fail before contact. The five weaker agents fail without making required contact at 9.1% to 18.5% of contact stages, versus 2.7% to 3.7% for the three strongest; 17% to 24% of their failed Standard and Realtime attempts reach no stage at all, versus 7% to 10%.

• Strong agents solve partly different tests. In Standard, no agent's solved set is a subset of another's; Claude Opus 5.5 and Gemini 4 Argon together solve 64 Standard tests that GPT-6 Astra fails. In Privileged, GPT-6 Astra solves 92% of Claude Opus 5.5's solved tests and 95% of Gemini 4 Argon's.

• The harness is part of what is measured. Switching harnesses changes success by up to 12 percentage points: GPT-6 Astra solves 34% in Codex versus 22% in Claude Code; Gemini 4 Argon solves 29% in Codex, 25% in Claude Code and 24% in Antigravity CLI; Claude Opus 5.5 solves 26% in Codex and 25% in Claude Code.

• Control approach depends on setting and resources. Across 1,489 sampled attempts, agents control the robot directly in 61% of Standard attempts but write programs in 82% of Privileged and 97% of Realtime attempts. In a soccer task, the GPU-enabled version led 10 of 12 Standard attempts to train a policy (7 of 8 in Realtime), and only trained policies won (three Standard wins, one Realtime); on the CPU version none of the 12 Standard and 8 Realtime attempts won.

Methodology in Plain English

The authors package each robotics task in the container format of Harbor, giving the agent a workspace where it can install packages, write programs and train models. Every task starts from a frozen initial scene and is graded by a verifier that sits outside the agent's container and that the agent cannot modify.

The central design choice is to constrain only the outcome, not the method. The agent gets a time limit, token budget, compute caps, a reset budget and, for some tasks, a step limit; within those, it decides whether to emit actions directly, write a planner or controller, or train a policy. The three settings then vary what the agent may see and do: Privileged runs the simulator in the same container so the agent can read full state and reset freely (its submitted trajectory is replayed separately for grading); Standard exposes the simulator as a service behind predefined APIs with proprioceptive and RGB-D access and no resets by default; Realtime removes the pause so the world advances while the agent thinks, or runs at a fixed fraction of real time such as 0.5x.

To find out where agents fail rather than just whether they fail, the authors split tasks into ordered stages with typed transitions and contact requirements, and have separate coding agents label the failure stage and phenomena of every failed attempt. Experiments cover eight agents (model plus harness, at high reasoning effort) on a 456-task core set — 901 tests — with one attempt per test.

Why This Matters

Impact on research. The paper argues that robotics benchmarking has been organized around visuomotor policy learning with a fixed interface, and that a different class of system — an unchanged general-purpose coding agent that reads an interface, picks a control strategy, executes and revises — needs its own evaluation. By separating what an agent can do with full simulator access from what it can do through sensors alone, EFE turns "state access" and "reset ability" into measured variables rather than confounds, and its stage-level labels give a diagnostic vocabulary beyond a single success number.

Real-world applications:

• Factory and warehouse automation, where an agent would have to write or adapt controllers for unfamiliar hardware and dynamics rather than deploy a fixed pretrained policy.
• Household service robots operating under partial observability and irreversible actions, where the Standard and Realtime settings approximate deployment conditions.
• Safety screening, as in the Embodied Safety Exam suite, which mixes hazardous with benign requests.
• Sim-to-sim engineering and controller retuning when a robot's dynamics change.

Industry relevance. The 12-percentage-point spread from harness choice alone shows that agent scaffolds, not just models, are a product-relevant variable; the result that only trained policies won the soccer task suggests that giving agents GPUs and training libraries materially changes outcomes. The same Standard interface is designed to transfer to physical hardware once outcomes can be graded, which the authors demonstrate in a preliminary buzz-wire game with a real robot arm.

Future Directions

• Repeated runs per test. Each agent currently makes a single attempt per test; multiple runs would separate repeated failures from run-to-run variance and yield confidence intervals for agent and setting comparisons.
• More Realtime tasks. Realtime has the largest share of tests that no agent solves (63% on the 54 dual-test tasks), and it is currently evaluated only on dynamic tasks due to compute constraints.
• Systematic real-robot evaluation. The Standard interface uses only sensor readings and the control interface, so it can be attached to hardware once the outcome can be graded; the buzz-wire demonstration is a first step.
• Accounting for evaluator bias. The core set was filtered using GPT-6 Luna, and most stage labels were written by GPT-6.1 Sol — both from the same family as the strongest agent — which the authors flag as a possible bias.

Target Audience

Researchers and engineers working on embodied AI, robot learning and LLM agent evaluation; benchmark designers who need to reason about state access, reset budgets and real-time constraints; and robotics practitioners assessing whether coding agents are viable as a control layer. The paper is also useful to agent-harness developers, since it treats the harness as part of the measured system rather than as an implementation detail.

Authors’ abstract

Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints. We therefore introduce the Embodied First Exam, a unified, comprehensive, and agent-native evaluation framework where coding agents can choose their own control approach: direct…