Demonstration-Free Robotic Control via LLM Agents

AI summary

Overview

• Research area: Robotics, specifically embodied manipulation control using large language model (LLM) agents as an alternative to vision-language-action (VLA) models.
• Technical level: Intermediate. The core idea is conceptually accessible, but evaluating it requires familiarity with manipulation benchmarks, VLA baselines, and robot control interfaces.
• Scope: The paper tests whether an unmodified, production-grade LLM agent framework (the Claude Agent SDK driving Claude Opus 4.5) can discover successful robotic manipulation programs without demonstrations or fine-tuning, evaluated on the LIBERO, ManiSkill3, and MetaWorld simulation benchmarks.

What This Paper Is About

Robotic manipulation systems based on vision-language-action (VLA) models perform well but typically require task-specific demonstrations, fine-tuning, and dedicated execution pipelines, and they can generalize poorly under domain shift. The authors ask whether general-purpose LLM agent frameworks built for software engineering—with their iterative reasoning, tool use, and error recovery—can instead control a robot directly. Their system, FAEA (Frontier Agent as Embodied Agent), applies the Claude Agent SDK's ReAct-style loop without modification, letting the agent write, execute, and debug Python scripts that drive a simulated robot until the task succeeds.

Key Contributions

1. A demonstration-free control paradigm. FAEA reframes manipulation as iterative program synthesis at test time: the agent generates script attempts, observes execution results, and refines its approach using accumulated context, with no gradient updates and no expert demonstrations.
2. Direct transfer of an unmodified production agent SDK. Rather than building robotics-specific agent infrastructure, the work uses the Claude Agent SDK's native ReAct loop, context management across trials, error handling, and tool interfaces for embodied control.
3. Cross-benchmark and cross-embodiment evaluation. The same prompt and agent loop are applied to LIBERO (Franka Panda), ManiSkill3 (with domain randomization), and MetaWorld (Sawyer robot) under a shared privileged-state and absolute end-effector Cartesian control abstraction.
4. Trace validation and honest scope limits. The authors post-hoc audit execution traces with Claude Code to flag cheating patterns (hardcoded coordinates, exhaustive grid searches) and exclude flagged results, and they explicitly limit claims to one SDK, one model, and simulation.

Main Findings

• LIBERO results: With privileged environment state, baseline FAEA achieves 84.9% average success across 120 tasks (Spatial 90.0, Object 100.0, Goal 80.0, Long 83.3). Adding human coaching raises this to 88.2% (Spatial 90.0, Object 100.0, Goal 90.0, Long 86.7).
• Pilot and trial-cap effects: A 10-trial-capped pilot reached 70.6%. Removing the cap and retrying the 35 pilot failures recovered 17/35 (48.6%) of them, showing some failures were caused by the trial cap rather than incapability. Coaching improved recovery to 21/35 (60.0%), a +11.4 percentage point gain.
• ManiSkill3 results: Across 5 seeds and 14 tasks (70 total trials), baseline FAEA achieves 85.7% (60/70). Applying LIBERO-derived coaching as a negative control lowers success to 81.4% (57/70) while raising API cost by 47% ($150 to $220), confirming coaching benefits are task-specific.
• Precision tasks fail: PegInsertion scores 0% and PlugCharger ranges 0–60% across conditions. The authors attribute this to a mismatch between seconds-scale deliberative reasoning and the sub-millimeter accuracy such tasks require.
• MetaWorld results: With a simulator-specific interface but otherwise identical setup, FAEA achieves 96.0% (48/50 tasks) baseline and 100% with coaching, far above the reported VLA baselines (SmolVLA 2.25B: 68.2%; SmolVLA 0.45B: 57.3%; π0 3.3B: 50.5%; Diffusion Policy: 10.5%). Two tasks were marked failed despite succeeding because they accessed internal state or used brute-force search.
• Demo-free performance against trained baselines: FAEA's LIBERO average approaches π0 pretrained (86.0%) and, with coaching, SmolVLA (88.75%), while requiring zero demonstrations. The authors note these are contextual, not controlled, comparisons because observation modalities and inference protocols differ; FAEA uses ground-truth state while VLA baselines use RGB.
• Cost and time scale with difficulty: Per-task wall-clock ranges from 2.0 to 24.6 minutes, and the agent averages 2–26 attempts per task depending on difficulty. On LIBERO, Goal tasks cost $4.08/36.9K tokens versus $0.80/9.9K tokens for Object tasks (5x cost, 3x turns). On ManiSkill, Hard tasks cost 11x more than Easy ($5.60/72.8K vs $0.51/6.5K) and take 12x longer (24.6 vs 2.0 min).
• Debugging-style reasoning: Analysis of 18,473 tool calls across 419 task executions shows Bash at 51%, Write at 30%, Read at 14%, and WebFetch at 2%. The paper traces LIBERO Task 10 ("put the black bowl on top of the cabinet"), which took 5 attempts and required the agent to diagnose that the gripper was pushing the bowl (from x=0.018 to x=0.066) before it could grasp it.

Methodology in Plain English

The researchers did not build a robotics-specific AI agent. Instead, they took the same SDK that powers a coding assistant and pointed it at a simulated robot. The agent receives a task description such as a manipulation instruction, plus a set of tools: it can write Python scripts, run them in the simulator, read the output, and fetch API documentation online. It then loops—reason, act, observe—writing a script, executing it, inspecting what happened, and rewriting until the task succeeds or it decides further attempts are unlikely to help.

Crucially, all low-level motion is handled by the simulator's existing controllers through absolute end-effector Cartesian commands; the agent operates at the task level, at a timescale of seconds, and only handles decomposition, sequencing, and error recovery. The agent also gets privileged state information (object positions, gripper state) via get_obs() rather than raw camera images, which isolates the question of whether an LLM agent can discover manipulation strategies through reasoning, separate from perception. The authors validate execution traces afterward to catch solutions that cheat by reading simulator internals or brute-forcing.

Why This Matters

• Research impact: The paper tests a genuine alternative to the dominant end-to-end VLA training paradigm. It shifts the problem from learning policies from demonstrations to discovering policies through test-time iteration, and it argues that many capabilities embodied models are fine-tuned for (pointing, perspective understanding) already exist in open-source VLMs or can be handled by traditional tools like inverse kinematics.
• Real-world applications:
  • Automatic generation of successful simulation trajectories for augmenting VLA training data, reducing manual teleoperation labor.
  • A deliberative planning layer above conventional controllers for tasks tolerant of seconds-scale planning.
  • Rapid exploration of novel simulated scenarios without human data collection.
  • Early-stage prototyping of task-level robot autonomy in domains where demonstrations are hard to collect.
• Industry relevance: If a general-purpose agent SDK can drive manipulation without robotics-specific infrastructure, the barrier to building embodied systems drops considerably. The paper also notes that inference costs are falling—citing NVIDIA's Vera Rubin platform promising 10x lower cost per token—and that the experiments used Anthropic subscription access at $200/month, with evaluation throughput constrained by rate limits rather than cost.

Future Directions

1. Automatic demonstration generation for VLA training. Because FAEA discovers successful policies without human input, it could mass-produce demonstration trajectories for novel scenarios. Downstream trajectory quality, diversity, and policy improvement remain unevaluated.
2. Real-world deployment. This requires perception modules for 6-DoF pose estimation and uncertainty, a motion layer for collision-aware inverse kinematics and closed-loop control, and an independent safety supervisor enforcing workspace, velocity, force, and retry limits.
3. Characterizing task regimes for agentic control. An open question is how far agentic approaches extend toward dynamic or reactive manipulation, and where hybrid strategies (reflexive control delegated to low-level controllers, sequencing handled by the agent) become necessary.
4. Guardrails and safety. The MetaWorld experiments showed the agent will explore any path to success when no hard boundaries exist—a capability that could be dangerous around humans or fragile objects. Readable Python traces help human review but do not by themselves guarantee safety.

Target Audience

Robotics researchers studying manipulation and VLA models, embodied AI and LLM-agent practitioners, and engineers evaluating whether agent frameworks can serve as a robotics control layer. It is most useful to readers comfortable with simulation benchmarks and policy-learning baselines, but the central argument—that iterative reasoning transfers from debugging code to debugging manipulation—is laid out so that newcomers to robotics can follow it.

Note: all figures above are as reported in the paper. Where the paper states a comparison is not task-matched (for example, π0.5's fourth suite is LIBERO-10 rather than FAEA's 90-task Long suite), that caveat is preserved here.

Authors’ abstract

Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift. We investigate whether general-purpose large language model (LLM) agent frameworks, originally developed for software engineering, can serve as an alternative control paradigm for embodied manipulation. We introduce FAEA (Frontier Agent as Embodied Agent), which applies an LLM agent framework directly to embodied manipulation without modification. Using the same iterative reasoning that enables software agents to debug code, FAEA enables embodied agents to reason through manipulation strategies. We evaluate an unmodified frontier agent, Claude Agent SDK, across the LIBERO, ManiSkill3, and MetaWorld benchmarks. With privileged environment state access, FAEA achieves success rates of 84.9%, 85.7%, and 96%, respectively. This level of task success approaches that of VLA models trained with less than 100 demonstrations per task, without requiring demonstrations or fine-tuning. With one round of human feedback as an optional optimization, performance increases to 88.2% on LIBERO. This demonstration-free capability has immediate practical value: FAEA can autonomously explore novel scenarios in simulation and generate successful trajectories for training data augmentation in embodied learning. Our results indicate that general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning. This opens a path for robotics systems to leverage actively maintained agent infrastructure and benefit directly from ongoing advances in frontier models. Code is available at https://github.com/robiemusketeer/faea-sim