Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena

AI summary

Overview

Research area: Embodied AI / robotics evaluation — specifically, benchmarking vision-language model (VLM) agents as general-purpose robot controllers (cs.RO).

Technical level: Intermediate. The paper is readable without deep robotics background, but familiarity with VLM agents, spatial reasoning benchmarks, and robot manipulation suites helps.

One-sentence scope: The paper introduces the Embodied Agent Arena — 1,000 evaluation cases spanning five capability domains — and uses it to test ten frontier VLM agents on whether they can go from perceiving a scene to actually satisfying every condition of a robot task.

What This Paper Is About

Robots powered by VLM agents are increasingly expected to look at a scene, decide what to do, and carry out a complete task — not just answer a question about an image. The paper asks how close today's best models are to that goal, and argues that average accuracy scores hide the answer. A model can locate an object and still fail to grasp it, or follow a path precisely and still miss its endpoint, so the authors build an evaluation that separates prediction accuracy, achieved subgoals, and final task success.

Key Contributions

1. An evaluation centered on complete task requirements. The arena scores prediction accuracy, functional grounding, subgoal achievement, and final task success across five capability domains, rather than reporting a single aggregate number.
2. A unified arena with a new geometric benchmark, GeoProbe. The 1,000-case arena combines 32 established sources with 168 GeoProbe cases (controlled Blender scenes plus real images) that vary camera motion, object motion, and scale references to isolate camera, pose, depth, displacement, and scale estimation.
3. A minimal, model-directed harness. Agents receive only source-native observations and operations plus a Python runtime (or the native manipulation agent loop); no extra perception or control modules are added, so task decisions stay with the model. Termination is scored independently of the agent's own claim of completion.
4. A diagnosis of failure after successful intermediate steps. Matched execution traces across ten models identify invalid endpoints, unmet goal conditions, wrong receiving regions, lost object state, and premature stopping.

Main Findings

• Closed-weight models lead all five domains, but with different profiles. Astra and Gemini 4 Argon place in the top two for Spatial Reasoning and Task Planning and the top three for Manipulation; Opus 5.5 leads Manipulation at 47.8% while trailing in Planning at 48.4%.
• Perceptual skill does not transfer to execution. Qwen-Max nearly matches Astra on Spatial Reasoning Pass@1 (68.9% versus 69.4%) but completes 36.9% versus 80.3% of Planning tasks. Qwen-397B, Qwen-27B, and DeepSeek-1017 reach 48.0–51.6% Spatial Pass@1 but only 19.1–31.2% Planning and 9.9–20.9% Manipulation success.
• Planning success is thinly shared. Only eight of 157 tasks are solved by all ten models, and seven of those belong to symbolic VirtualHome; 39 tasks have only one or two successful models, including twenty ALFRED or ALFWorld tasks. Astra solves 126 tasks, adding Gemini 4 covers 137, and all ten together cover 142 — one extra model supplies eleven of the sixteen additional successes.
• Aggregate rankings depend on task mixture. Opus 5.5 leads Gemini 4 in Manipulation by 47.8% to 44.0% over assignments, but equal-weighting thirteen sources reverses the order to 44.5% versus 47.4%; RoboTwin alone supplies 50 of 182 assignments and favors Opus (23 versus fourteen successes).
• Similar geometric error can have different causes. On GeoProbe depth, Gemini 4 and Qwen-397B have similar raw AbsRel (0.299 and 0.312), but after oracle scale alignment the errors are 0.065 and 0.203 — Gemini 4's error is largely a common scale bias, while Qwen-397B retains shape distortion. Gemini 4's median predicted/true depth ratio is below one in all sixty case–run observations. Astra combines 96.3% ordinal accuracy with 0.104 raw AbsRel, falling to 0.038 after alignment.
• Camera sensitivity is not the same as a motion-detection floor. Astra and Gemini 4 miss none of fifty moving observations under a fixed camera but miss 26 and 37 under a large view change, without falsely flagging stationary controls. Qwen-27B misses all fifty moving observations in every camera condition.
• Valid endpoints do not guarantee physically feasible paths. Astra hits both image endpoint regions on 35 of 56 TraceSpatial paths, yet 21 of those fail a 3D boundary condition; Gemini 4 reaches 45 image-valid paths with sixteen such failures but has ten collision-only failures versus Astra's one.
• Region overlap and action-point validity are separate requirements. Across 350 case–run predictions, Qwen-397B passes the point criterion alone 96 times but both point and box only eighteen times. Gemini 4 shows the reverse pattern sixteen times: a scissors-grasp case has rectangle IoU of 0.784 while the chosen point misses the grasp mask, repeating in four of five runs.
• The right operation can target the wrong receiving region. On a matched ALFRED soap task, DeepSeek cleans the soap and places it in Bathtub, ending with 1/3 goal conditions; Gemini 3.8 later switches the destination to BathtubBasin and reaches 3/3, and Qwen-Max also changes the receiving region.
• Issuing an action is not the same as reaching the state. On HumanCLAW sitting, Astra and Gemini 4 record proximity within 20 cm and a sit action in five and seven episodes respectively, yet there are no recorded successes; Gemini 4 and DeepSeek declare themselves seated and stop after 44 and 70 rounds, both failing the private native verifier.
• Long manipulation sequences fail late. Fable, Sol, Gemini 3.8, Astra, and Opus 5.5 reach CALVIN's first goal in seven or eight of ten episodes but the third goal in zero, one, four, seven, and four; across all models only nine of eighteen episodes reaching four goals finish the fifth.
• Physical progress can be lost and must be rebuilt. On CALVIN sequence 0526, DeepSeek remains on block retrieval during rounds 17–27 (block height returns to about 2 cm after a carrying attempt) and starts stacking at round 28, versus round 11 to 13 for Opus; both finish under the same 40-round allowance, using 37 and eighteen rounds.
• Converging coordinates do not certify a grasp. On PlaceSphere, DeepSeek brings the gripper within 2.8 mm of its hover target, yet four later checks return is_grasped=false; Opus confirms a grasp and satisfies the on-bin and static conditions. Opus completes PickCube, PlaceSphere, and StackCube but fails PegInsertionSide and PlugCharger, whose criterion requires 5 mm positional and 0.2 rad angular accuracy.
• Feedback helps only when it resolves the blocking requirement. Qwen-27B's self-review on eighteen Spatial cases fixes one wrong answer but breaks two correct ones, and crop review fixes two but breaks four; in Affordance, self-review repairs six of nine out-of-bounds boxes (mean IoU 0.034 to 0.152) yet none reaches the 0.5 criterion. Tool-assisted review yields a lower mean IoU of 0.145 but one accepted box, while Astra also gains one.
• Code repair restores execution without completing tasks. Qwen-Max records execution errors in 103 of 200 CALVIN rounds including 95 syntax errors; of 65 episodes with an execution error, 62 later execute code without error, but only three subsequently pass the native verifier. Among thirty error-free episodes with an observed native prefix, 24 fail.
• Failed-token cost is structured differently across models. HumanCLAW supplies 13 of 157 Planning tasks but 81.7% of Astra's failed tokens (its eight sitting tasks alone supply 63.7%); excluding it, Astra's failed-token share drops from 55.5% to 19.4% and Gemini 4's from 64.0% to 33.5%. Shares remain 55.3–67.8% for other closed-weight models and 73.6–85.8% for the three open-weight VLMs.
• Token cost is dominated by repeated input processing. In seven HumanCLAW traces, 98.23–99.46% of reported tokens are prompt tokens; Gemini 4's unresolved plant search records 1.95M tokens across 73 executed rounds.
• Resource ceilings differ in kind. DeepSeek's ScienceWorld animal search exhausts decision rounds with 89 environment steps unused, while Gemini 4's BEHAVIOR-1K toolbox attempt is blocked by the 640k-token ceiling after 37.5 minutes of a 180-minute allowance. On matched chair navigation, Gemini 4 uses seventeen rounds and 158k tokens versus DeepSeek's 65 and 938k, yet takes 437 versus 406 seconds.

Methodology in Plain English

The authors built a single test arena with 1,000 cases drawn from 32 existing robotics and vision benchmarks, plus 168 new cases from GeoProbe, a benchmark they created in Blender with controlled variations in camera motion, object motion, and scale reference, supplemented by real image pairs. Cases are organized into five domains: Geometry (estimating camera parameters, pose, depth, displacement, and scale), Spatial Reasoning (relations, viewpoints, and reference frames from images and video), Affordance (where to touch or act on an object), Task Planning (composing supplied operations to satisfy compound goals), and Manipulation (actual robot arm and gripper control).

Ten VLM agents are evaluated under a deliberately thin harness. The models get the same observations and operations the original benchmark provides — nothing more — and must decide for themselves what to do, with a Python runtime for four domains and the native agent loop for manipulation. Importantly, the evaluator checks the submitted answer or the environment state independently of what the agent claims; declaring a task complete earns no credit. Fixed-input questions are run five times per case for nine models and three times for Fable, while tracing, polygon outputs, planning, and manipulation use one episode per assignment. Episodes cut off by API or execution errors count as failures.

Beyond scores, the authors perform diagnostic analyses on matched cases: comparing raw versus scale-aligned depth error, replaying the same ALFRED or CALVIN task across models, and pairing self-review or tool-assisted review rounds with the answers that changed. This trace-level evidence is what lets them distinguish a perception error from a grounding error from a stopping misjudgment.

Why This Matters

Impact on research. The paper argues that leaderboard averages are the wrong unit of analysis for embodied agents. By showing that identical error magnitudes can stem from scale bias versus shape distortion, that region overlap and point validity diverge, and that valid code execution can leave a task incomplete, it gives the field a vocabulary for diagnosing which requirement failed. It also shows that aggregate rankings flip when tasks are reweighted by source, which is a caution for how embodied benchmarks are reported.

Real-world applications:

• Household and service robotics — the ALFRED-style findings (placing soap in the tub instead of the basin) map directly to failure modes in home assistants, where the correct action aimed at the wrong sub-region is a common and costly error.
• Warehouse picking and packing — the PlaceSphere result (gripper within 2.8 mm yet no grasp) is exactly the margin-of-error problem in bin picking and kitting.
• Laboratory and scientific automation — the ScienceWorld and Archaeology Dating traces show agents wasting limited decision rounds on repeated inspection instead of acting, relevant to lab automation with strict step budgets.
• Assistive and care robotics — the sitting-task traces, where an agent approaches a couch, issues a sit command, and declares success without the required body–object relation, illustrate safety-relevant false completion in human-proximate settings.

Industry relevance. For teams building VLM-driven robot stacks, the paper indicates that scaling perception quality alone will not close the execution gap, that open-weight models remain far behind on Planning and Manipulation (19.1–31.2% and 9.9–20.9% respectively), and that token budget, decision-round budget, and wall-clock time are separate constraints — a model can run out of any one of them, and fixing one does not fix the others.

Future Directions

• Memory across sessions is not evaluated. Table 1 marks memory as not evaluated for the arena, leaving open how agents should carry state across episodes rather than within one.
• Diagnosing the private verifier's unmet condition. In the sitting traces, the authors note that the trace establishes a stopping misjudgment but the private verifier does not identify which pose or contact condition remained unmet; making those conditions inspectable is an open problem.
• Turning trace evidence into targeted repair. The paper shows feedback helps when it restores a missing prerequisite (a receiving-region correction, a confirmed grasp) and hurts when it revisits already-correct judgments; how to build agents that select the right repair is unresolved.
• Budget-aware agent design. Since failed-token expenditure concentrates in a few generously budgeted sources for the strongest planners but spreads broadly for weaker models, and since different resources (rounds, tokens, wall time) terminate different attempts, allocating search and deciding when to stop remain open design questions.

Target Audience

Researchers and engineers working on embodied AI, VLM-based robot control, and agent evaluation; benchmark designers who need to justify aggregate scores; and robotics product teams deciding whether a frontier VLM agent is ready for a given deployment. Readers new to the area can follow the plain-language framing of the five domains and the trace examples, while specialists will find the diagnostic breakdowns — scale-aligned depth error, point-versus-region validity, code repair versus task completion — most useful.

Authors’ abstract

A robot generalist must interpret a scene, act on it, and satisfy every condition of a task. How far do frontier vision-language models (VLMs) meet these demands? We introduce Embodied Agent Arena, evaluating ten VLM agents across Geometry, Spatial Reasoning, Affordance, Task Planning, and Manipulation. Its 1,000 cases combine 32 established sources with GeoProbe, our 168-case benchmark for camera, motion, depth, and scale estimation. A minimal harness leaves task decisions to the model through source-native interfaces. Closed-weight models lead all five domains, whereas open-weight VLMs show useful spatial reasoning but substantially lower planning and manipulation success. Beneath these averages, similar geometric errors conceal different scale and shape distortions, and valid region estimates need not identify usable action points. During execution, agents can achieve early subgoals yet lose object control, bind an action to the wrong receiving region, or stop before the required physical state holds. Additional computation does not uniformly resolve these failures: code repair can restore execution without completing the task, while self-review can reverse correct judgments. Failure costs also differ: the strongest planners concentrate failed-token expenditure in prolonged body-control tasks, whereas other models fail broadly across procedures. Matched traces show how successful recovery uses feedback to restore a missing prerequisite and advance the remaining task.