RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory

AI summary

Overview

Research area: Robot learning / embodied AI — specifically vision-language-action (VLA) policies, LLM-driven code optimization, and agent scaffolding ("harnesses") for robotic manipulation.

Technical level: Intermediate. The core idea is easy to grasp, but full appreciation requires familiarity with VLA policies, manipulation benchmarks, and agent-design loops.

Scope (one sentence): The paper introduces RoboArchitect, a system in which a coding model automatically writes and iteratively repairs the control program wrapped around a frozen robot policy, and shows across multiple benchmarks and policies that this raises held-out manipulation success without any weight updates.

What This Paper Is About

Robot policies (VLAs) are usually adapted to a new robot or task by collecting demonstrations and finetuning the weights. Recent work shows that simply wrapping a frozen policy in hand-written control code — an "agent harness" that decides when to call the policy, when to use vision tools, and when to command the arm directly — can work much better, but these harnesses are engineered by hand, one per setup. RoboArchitect asks whether that entire engineering loop (run the robot, watch failures, diagnose, rewrite the code, retest) can be automated by a coding model, so that a frozen general policy can be turned into a specialist without touching a single weight.

Key Contributions

1. An automated harness-optimization loop. A coding model observes full rollout traces from a frozen policy, diagnoses failure modes, and proposes revised harness code; a revision is kept only if it improves performance on the optimization episodes, giving a monotonic improvement trajectory (capped at 5 iterations).

2. A demonstration that harness design substitutes for finetuning. Across benchmarks and policies, evolved harnesses raise held-out success from 9.7% to 43.3% on unseen scenes and from 12.7% to 40.2% on unseen tasks, and in several setups a harness around a general policy outperforms the finetuned version of the same policy.

3. Autonomously evolved persistent memory. The harness is free to invent its own memory schema — no rules are specified — and it discovers data structures recording grasp heights, grip styles, task types needing a hold, scene-reset periods and per-robot finger-closing directions, which are actively read during held-out episodes.

4. A single unified harness spanning multiple embodiments. One harness evolved over a procedurally generated training pool across LIBERO-Pro, VLABench, SimplerEnv WidowX and SimplerEnv Google adapts at runtime to Franka Panda, WidowX and the Google mobile manipulator, becoming a pure code-as-policy agent when wrapping the general X-VLA and preserving native competence when wrapping finetuned X-VLA specialists.

Main Findings

• Large held-out gains without weight updates. Mean over all setups: unseen scenes 9.7% → 43.3% (+345%, described as 4.5×), unseen tasks 12.7% → 40.2% (+216%, described as 3.2×). The mean over cross-benchmark and out-of-domain setups alone goes 7.3% → 58.4% (+696%) on unseen scenes and 6.3% → 48.7% (+671%) on unseen tasks.

• The harness can beat finetuned weights. With a general X-VLA on VLABench, the harness reaches 52.6% on unseen scenes and 72.5% on unseen tasks, versus 35.2% and 32.1% for the X-VLA checkpoint finetuned on VLABench. On SimplerEnv WidowX unseen scenes it reaches 87.5% versus 70.8% finetuned; it falls short only on SimplerEnv WidowX unseen tasks (66.7% versus 87.5% finetuned).

• Iteration matters, not just one-shot design. The one-shot harness (iteration 1) averages 21.4% on unseen scenes, and iterative refinement lifts this to 43.3%; some one-shot designs fall below the bare policy while the iterated harness beats it on nearly every held-out set. The exception is RLDX-1 on RoboCasa-365, which stays roughly flat (27.9% bare → 27.8% iterative on unseen scenes, −0.2) and only improves on its optimization episodes.

• The harness learns its own policy/code division of labor. With general X-VLA on SimplerEnv WidowX, the policy commanded a fully open gripper at every step, so the harness learned to disregard it entirely and became a vision-based code-as-policy agent — 0% to 100% on optimization episodes, 87.5% on unseen scenes and 66.7% on unseen tasks. With finetuned X-VLA specialists, the unified harness instead keeps the VLA in charge and intervenes only when it fails (e.g., taking over via the pointing tool if the gripper has never approached the instructed object).

• Interpretable, failure-specific control routines. Evolved routines include lifting a jammed arm clear by monitoring commanded-versus-actual gripper motion, computing horizontal approach vectors when depth maps show overhead obstructions, rescaling action magnitudes to close a toaster-oven door that requires closure within 0.5% of its opening range, tethering a drifting mobile base to workspace anchors, and redirecting to the named object (a moka pot) instead of a memorized location.

• Unified harness results. Around the general X-VLA it lifts the mean over all benchmarks from 1.8% to 41.6% on the training pool (+2228%) and from 1.9% to 28.8% on held-out episodes (+1417%). Around finetuned specialists it moves 58.0% → 60.7% on the training pool and 45.9% → 47.0% on held-out.

• Memory is read often and is usually useful. Reads per held-out episode and share of actionable reads: X-VLA on VLABench 3.4 / 98%, X-VLA on SimplerEnv WidowX 3.5 / 96%, MolmoAct-2 on LIBERO-Pro-10 9.9 / 99%, π0.5 on RoboCerebra 13.0 / 95%, RLDX-1 on RoboCasa-365 11.0 / 92%. Example learned entry: closing at 18% of object height (raised to a 0.8 cm minimum) lifted nothing on the carrot, while 45% of object height (1.4 cm on the carrot, 3.9 cm on the eggplant) lifted both.

• Per-robot timing adaptation. Replacing fixed time budgets with arrival-based termination raised training-pool success on the slower 3 Hz Google robot from 11.0% to 30.2%.

Methodology in Plain English

The starting point is a "bare" harness that simply forwards camera images, depth, gripper pose and status, plus the language instruction, to a frozen VLA and executes whatever actions it returns. The researchers then split each benchmark's episodes into a training portion (used only to optimize the harness) and held-out portions, evaluated under two protocols: unseen scenes, where tasks are the same but arrangements differ, and unseen tasks, where entirely different tasks test skill-level transfer.

The optimization loop mirrors a human engineer. Every evaluation logs complete traces — per-step frames, robot readings, and all model and tool outputs. A coding model (claude-opus-5, via the Claude Code client at maximum effort) reads those traces, diagnoses why episodes failed, and writes a candidate revision of the harness. The candidate may call the policy, invoke auxiliary tools, or command the arm directly. It is re-run on the same training episodes and kept only if it beats the current harness, for at most 5 iterations.

The available tools are pretrained off-the-shelf models: Qwen3-30B-A3B-Instruct as the language model; Molmo2-8B or Qwen3-VL-8B-Instruct as the vision-language model; Grounding-DINO for detection; SAM 3 for segmentation; Molmo2-8B for pointing; and GraspGen for grasp poses, plus geometric utilities that lift pixel-depth pairs into 3D base coordinates. Critically, the coding model is never told how to divide control between policy and code, nor what the memory should store — both emerge from the rollout evidence. For the unified harness, the training pool consists of procedurally generated episodes with new scene arrangements, objects, instructions or tasks, evaluated afterwards on the benchmarks' own standard episodes.

Metrics are standard: the official progress score on VLABench (partial credit for grasping the right object and each goal condition met) and success rate averaged over tasks elsewhere; for simplicity all are called success rates. Step limits are matched across all compared numbers, including finetuned references.

Why This Matters

The paper reframes where the performance headroom in robot manipulation sits: not only in policy weights, but in the agent program wrapped around them. If a coding model can recover that headroom automatically, adapting a robot to a new lab, task suite, or embodiment becomes a matter of running rollouts rather than collecting demonstrations.

Real-world applications:

• Fast redeployment of existing robot fleets — adapting a shipped policy to a new workspace layout or object set without new demonstration data.
• Warehouse and lab automation, where scene arrangements and object locations drift but the underlying pick-and-place skills stay the same.
• Multi-robot deployments where one harness must span arms with different speeds, gripper geometries and action layouts.
• Inspection and debugging of learned policies — because each revision is readable code targeting a named failure mode, engineers can audit exactly why an agent behaves a certain way, unlike weight updates.

Industry relevance: the result that a frozen general policy inside an evolved harness can exceed a benchmark-finetuned checkpoint on VLABench and SimplerEnv WidowX unseen scenes suggests a cheaper, more inspectable alternative to per-customer finetuning pipelines, and connects directly to the current practice of building agent scaffolding around foundation models.

Future Directions

• How many iterations are worth running? All results use a fixed cap of 5 optimization iterations; training-set performance improves consistently across iterations, but where returns saturate is unexplored.
• Richer contact dynamics. Extending the approach to dexterous in-hand manipulation or tool use, where the space of failure modes is much larger than in pick-and-place, and seeing what routines and memories evolution discovers there.
• Sim-to-real transfer. All evaluations are in simulation; because the harness reads only standard inputs (camera images, depth, the robot's own readings), the interface should transfer, but testing evolved harnesses on real robots — and how many further iterations a real deployment needs — is future work.
• Cross-embodiment generality. Whether a single unified harness can be pushed further, since the finetuned-specialist version of it currently yields only modest gains (58.0% → 60.7% on the training pool) because those policies are already strong.

Target Audience

Robotics and embodied-AI researchers working on VLA policies, policy adaptation, and agent architectures; practitioners building LLM-driven code-generation loops for control; and engineers who need to deploy a pretrained manipulation policy on new hardware or scenes without a finetuning pipeline. It is also useful for readers interested in emergent memory structures in agents, since the memory is designed by the model rather than specified by the authors.

Authors’ abstract

Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. We show that harness design can be automated, offering a strong alternative to finetuning on a novel setup. In RoboArchitect, a coding model observes a robot's rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of episodes for its optimization and is evaluated on held-out scenes and tasks. Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies. The gains are largest where the benchmark is far from the policy's training data: wrapping a general-purpose policy, the evolved harness yields a specialist that often outperforms even the finetuned version. Intuitive control strategies are automatically designed, such as lifting a jammed arm clear or reaching in horizontally when an obstacle blocks the way from above. The harness also evolves its own persistent memory, carrying learned experience, such as effective grasp depths, into later decisions. These results suggest that substantial headroom in robotic manipulation lies in the agent harness, and its design can be effectively automated and optimized without weight updates.