Coding Agents for Generalized Task and Motion Planning Problems

AI summary

Overview

Research area: Robotics, specifically task and motion planning (TAMP), generalized planning, and the use of LLM-based coding agents for synthesizing robot control programs.

Technical level: Intermediate. The high-level question (can coding agents write reusable robot programs?) is accessible, but the paper assumes familiarity with Markov decision processes, symbolic planning, and motion-planning terminology.

One-sentence scope: A large-scale empirical study comparing off-the-shelf coding agents (Claude Code with Opus 5, and Codex with GPT-5.6 Sol and GPT-6 Astra) against hand-engineered TAMP planners and LLM-based generalized planning baselines on 28 simulated robotic environments, totaling 98,000 evaluation episodes.

What This Paper Is About

Task and motion planning problems require a robot to make discrete choices (which object to move, which tool to use, which subgoal to pursue) that are tightly coupled to continuous geometric, kinematic, and dynamic constraints. Generalized TAMP tries to exploit regularities across problem instances so that new instances become easier, but existing methods require substantial domain-specific engineering such as hand-designed predicates, operators, samplers, and skills. This paper asks whether general-purpose coding agents, given only a task description and simulator access, can write a program that generalizes across instances and solves the task without any of that TAMP-specific scaffolding.

Key Contributions

1. A large-scale systematic study of coding agents for generalized TAMP. The study covers 28 simulated environments, seven program synthesis methods, five runs per method per environment, 100 held-out test instances per resulting program, and 98,000 evaluation episodes in total. Each synthesis run has a budget of $20 in model usage.

2. An evaluation protocol that separates synthesis from execution. The agent has a fixed synthesis budget and simulator access during learning, may run experiments and rewrite code freely, but returns a single program that is then frozen and evaluated on unseen instances with no LLM involved at test time. In the main setting the agent receives no environment source code and no hand-designed planning abstractions.

3. Comparison against strong baselines. Results are compared with TAMP planners that combine symbolic search with sampling (available for 16 of the 28 environments), with LLMGenPlan (a re-implemented LLM-based generalized planning method using Opus 5, given full environment source code), and with a one-shot generation variant.

4. Qualitative analysis and open release. Analysis of synthesized programs and interaction logs, including unexpected manipulation strategies; release of all code, including the full prompts given to the agents.

Main Findings

• Coding agents beat hand-engineered planners. On the 16 environments where a planner is available, all three agent configurations outperform the planners: Claude Code averages 82% success, 1.7 times the planner's 47%, while Codex averages 95% with GPT-6 Astra and 56% with GPT-5.6 Sol. Across all program synthesis methods this corresponds to mean success of 56% to 95% versus 47% for the planners.

• Per-family success rates. Astra averages 99% success on Kinematic2D, 97% on Dynamic2D, 93% on Kinematic3D, 65% on the harder Dynamic3D, and nearly 100% on PDDLStream. Opus follows with 96%, 92%, 90%, 45%, and 78% respectively. Agents exceed the planning baseline in 15 of 16 environments with a planner (Astra), 12 (Opus), and 9 (Sol).

• Agents beat LLM baselines. All three agents outperform one-shot generation and LLMGenPlan in mean success over all 28 environments.

• Efficiency improves with scale. As object counts grow, the agents' programs maintain higher success than the planner while using an order of magnitude less computation per instance on average. On the 14 environments with a planner and multiple object counts, Opus and Astra programs take 2.1 s and 0.5 s per instance versus 29 s for the planners. Astra reaches 1.3 ms per action on average, about 9 times faster than Opus at 11.7 ms.

• Unexpected strategies were discovered. Programs found non-prehensile maneuvers and uses of environment layout not previously published for these benchmarks, such as rotating and regrasping a tool from the left side to scoop more balls at once in Dynamic2D ScoopPour, or ignoring the sweeper tool in SweepIntoDrawer and using the gripper to move cubes one at a time.

• Interaction supports model building and calibration. In Shelf, one Opus run built an initial kinematic model of the Kinova Gen3 arm from its own knowledge, probed the arm with a grasped cube, and fitted six parameters describing robot mount and grasp offsets, reducing RMSE between predicted and observed cube positions from 38.9 to 1.8 mm. In BalanceBeam, one Sol run moved small-block placement targets closer to the beam's center, raising success from 6% to 64%.

• Agents write custom tests to find edge cases. In StickButton, Opus repeatedly called reset with different seeds to find wall-adjacent sticks and high buttons, then combined edge cases with typical instances into a diverse test suite.

• Iterative revision produces large jumps. In one Sol Blocked run, a commit adding a spare-block fallback raised success from 15% to 56%, with 41 newly solved instances and no lost successes; subsequent commits reached 83% final success, solving all 80 instances with an alternative block but only three of the 20 without one. Sol also outperformed the other agents there (75% versus 38% for Opus and 74% for the planner).

• Astra is the most consistent, Opus is not far behind. Astra achieved higher mean success than Opus in 20 of 28 environments, but in 11 of those 20 the best Opus program scored at least as high as the best Astra program, so Astra's edge comes largely from consistency. Opus was stronger than Sol in 22 of 28 environments, with four ties.

• Source code access helps but is not sufficient. Giving agents the environment source code raised mean success from 74% to 84% for Opus and from 86% to 95% for Astra. LLMGenPlan also receives the source code with the same Opus 5 model and $20 budget but reaches 28%, while Opus + source is higher in 27 of 28 environments. In SortClutteredBlocks, source access let programs read each cube's target bin, raising success from 43% to 95%; in SweepIntoDrawer, where Opus and Sol score zero in the main setting, source access raised Opus's mean to 57% with one run at 97%. Source access also produced slower programs (Opus + source 42.5 ms per action; Astra + source 50.4 ms), because many invoke environment code at decision time.

• Failures remain. Some dynamic three-dimensional environments are still largely unsolved in the main setting, particularly ones requiring sweeping or pouring many small objects.

Methodology in Plain English

For each environment, the researchers handed a coding agent a written task description plus access to a simulator through two functions: one to sample a starting state and one to execute an action. The agent ran inside a sandboxed Docker container with a separate filesystem, no network access, and only a bare Python interpreter with NumPy and SciPy, so it had to write end-to-end code rather than lean on existing libraries. No hand-written TAMP predicates, operators, samplers, or skills were provided.

Within a fixed budget, each agent decided for itself what to run: it could write and execute test scripts, probe the simulator, render states as images, and revise its program. It had to deliver the policy as a class with a reset method for episode initialization and a get_action method for computing each action, and it committed each version to a git repository so every revision could be replayed. After synthesis, the program was frozen and evaluated on 100 held-out instances per environment, with seeds generated randomly so that agents were unlikely to have tested them, using the 60-second per-instance timeout from the original KinDER protocol. An additional "+ source" setting let some agents read the environment implementation and import helpers such as inverse kinematics solvers, serving as a reference for performance with complete environment knowledge. Baselines were TAMP planners given the benchmarks' hand-designed components, a re-implemented LLMGenPlan run with Opus 5 with chain-of-thought prompting and thinking disabled, and the first LLMGenPlan program as a one-shot baseline.

Why This Matters

Impact on research: The results suggest that off-the-shelf coding agents can reduce much of the domain-specific engineering that TAMP methods currently require, and establish them as a strong baseline for generalized TAMP research. The discovery of manipulation strategies absent from any published solution for these benchmarks is hard to attribute to memorized training data, which speaks to genuine agentic physical reasoning. Notably, the environment setup assumes fully observed, object-centric states and simulator access during synthesis, so the paper does not establish performance under perception uncertainty; the authors also note that the models' training data are undisclosed, so prior exposure to benchmark code cannot be excluded.

Real-world applications:

• Warehouse and logistics robots that must retrieve, pack, or rearrange objects among obstacles with varying object counts.
• Industrial assembly, where a robot must decide which grasp, tool, or placement strategy to use under tight geometric constraints.
• Household manipulation, including non-prehensile behaviors such as sweeping, pouring, and tossing that the agents discovered on their own.
• Rapid prototyping of robot behaviors: instead of hand-engineering a planner for each new task, a developer could describe the task and let an agent synthesize and test a program against a simulator.

Industry relevance: The synthesis budget of $20 in model usage per run and the agents' low per-action computation times (as low as 1.3 ms) point to a practical workflow in which a coding agent generates reusable robot code once, and the deployed program runs cheaply without any LLM at test time. This is directly relevant to robotics companies weighing whether to build domain-specific planners or to leverage general-purpose coding agents plus simulation.

Future Directions

• Extend beyond fully observed, object-centric states. The current setup assumes full observability and simulator access during synthesis, so testing under perception uncertainty is an open question.
• Improve the hardest dynamic three-dimensional tasks. Environments requiring sweeping or pouring many small objects remain largely unsolved in the main setting.
• Investigate source-code access trade-offs. Source access raised success rates but made programs slower on average, sometimes by nearly 200 times in individual environments, so understanding when to expose implementation code versus force distillation into self-contained programs is an open design question.
• Rule out training-data contamination more rigorously. Since model training data are undisclosed, the authors rely on interaction logs and novel strategies as evidence against mere recall; stronger controls would strengthen this claim.

Target Audience

Robotics and embodied-AI researchers working on task and motion planning, generalized planning, and program synthesis; practitioners evaluating whether coding agents can replace hand-engineered planning components; and machine learning researchers interested in how LLM-based agents perform on physical reasoning tasks where prior work has shown LLM calls alone perform poorly.

Authors’ abstract

Task and motion planning (TAMP) problems remain difficult even with full observability and object-centric states because discrete decisions are tightly coupled to geometric, kinematic, and dynamic constraints. Generalized TAMP addresses this difficulty by exploiting regularities across problem instances to reduce planning effort on new instances. However, existing methods require substantial TAMP-specific engineering. We investigate whether coding agents can automate this process by synthesizing programs that generalize across instances. Given a task description and simulator access, each agent chooses how to interact with the environment while developing a program within a fixed synthesis budget. The program is then frozen and evaluated on unseen instances. We evaluate Claude Code (Opus 5) and Codex (GPT-5.6 Sol and GPT-6 Astra) on 28 simulated environments from KinDER and PDDLStream, with object counts beyond those evaluated in the original benchmark. Across all program synthesis methods, we evaluate 980 generated programs on 100 held-out instances each, 98,000 evaluation episodes in total. Overall, we find that coding agents are surprisingly effective at generalized TAMP: all three agent configurations outperform hand-engineered planners, one-shot generation, and an LLM-based generalized planning baseline in mean success (56% to 95% versus 47% for the planners, on the 16 environments where a planner is available). As object counts grow, the agents' programs maintain higher success than the planner, using an order of magnitude less computation per instance on average. Logs show agents using interaction to calibrate physical models, test edge cases, and refine strategies. We release all code, including the full prompts given to the agents. These findings suggest that coding agents are a strong baseline for generalized TAMP.