Co-Evolving Robot Orchestrators and Policies through Deployment

AI summary

Overview

Research area: Robotics / robot learning — deployment-time self-improvement of agentic robot systems that combine vision-language-action (VLA) policies with vision-language-model (VLM) orchestrators.

Technical level: Advanced. The paper assumes familiarity with VLA policies, LoRA fine-tuning, flow-matching training objectives, language-model planning agents, and episodic memory for embodied agents.

Scope: The paper proposes Robo-COP, a system that co-evolves a robot's orchestrator and its VLA policy during deployment, and evaluates it on ten simulated RoboLab tasks and three real-world DROID tasks.

What This Paper Is About

Robot policies trained on large datasets still fail under the distribution shift they meet once deployed, and agentic harnesses that wrap such policies in a language-model orchestrator can only route around those failures, never fix them, because the policy stays frozen. Fine-tuning the policy during deployment is hard in three specific ways: useful data must come from task attempts rather than a separate exploration phase, fine-tuning does not guarantee improvement, and updating the policy alone leaves behind an orchestrator whose memory was built around the old checkpoint. Robo-COP addresses this by letting the orchestrator and the policy improve together, purely from the robot's own deployment executions.

Key Contributions

1. Co-evolving orchestrator and policy. The paper frames deployment-time learning as improving an agentic harness and its policy together, rather than adapting how a fixed set of capabilities is used.

2. Agentic data curation. Deployment executions are turned into skill demonstrations without manual labels: episodes are segmented into skills, and successful skills are kept even from episodes that ultimately failed. Training is triggered only when the accumulated data warrants it.

3. Skill verification and memory revision. Each fine-tuned candidate policy is verified at the skill level before adoption, including checks for regressions on skills admitted by earlier candidates, and policy-dependent memory entries are revised after every adoption or reversion.

4. Performance gains in simulation and the real world. Robo-COP raises mean held-out success from 64.8% to 73.8% on RoboLab simulation, and from 38.3% to 50.0% on three real-world DROID tasks, over a frozen-policy harness.

Main Findings

• Held-out success improves with co-evolution. Robo-COP reaches 73.8% mean held-out success over ten simulated tasks, versus 64.8% for the frozen-policy baseline and 65.8% for fine-tuning on a fixed schedule without verification. The fixed schedule recovers only 1.0 of the 9.0 points.

• Gains concentrate where the policy actually changed. On the six tasks where Robo-COP adopted an update, it averages 11.3 points over the frozen-policy baseline; on the four tasks where it kept the base policy, the gap is 5.5 points, where the systems differ only in memory and deployment histories.

• No single task drives the result. Omitting any single task leaves a mean gain of 4.0–12.0 points over the frozen-policy baseline and 4.9–10.7 points over the fixed schedule.

• Regimes separate the methods. Where the VLA alone suffices, frozen and fixed harnesses reach 89.3%, Robo-COP 80.0%, and ASPIRE 66.0% (ASPIRE cannot call the VLA). Where scripted primitives suffice, ASPIRE leads at 73.5% and Robo-COP is close at 72.0%, versus 55.0% frozen and 50.0% fixed. Where neither suffices, ASPIRE falls to 16.0% while Robo-COP reaches 70.0%, 16.7 points above the frozen-policy baseline.

• Fine-tuning improves grasping most. Robo-COP fails 131 of 500 held-out trials versus 176 for the frozen-policy baseline. Grasp or drop is the dominant failure in both (40 and 68 failures), and accounts for 28 of the 45 fewer failures. Every first training request targeted a pickup skill.

• The controller decides sensibly when to train. It trained on the seven tasks where the harness still failed often (frozen baseline succeeding on 45–79% of deployment trials) and waited on the three where it rarely failed (80–96%). On those three tasks the fixed schedule's nine training runs bring no net gain over the frozen baseline.

• Verification rejects bad candidates. Verification rejected 5 of 12 candidates. On the four tasks whose first candidate was reverted, the fixed schedule (which deploys every checkpoint) falls below the frozen-policy baseline on three. Rejection does not end learning: on three of those four tasks the next candidate was adopted.

• Robo-COP tracks a deployment example. On Recycle cup clutter, the controller requests training after trials 45 and 75 and adopts the candidates at trials 70 and 100; success reaches 76% over the final 25 deployment trials, versus 60% for the frozen baseline and 52% for the fixed schedule.

• Learning does not cost deployment success. Robo-COP succeeds on 73.2% of 100 deployment trials, versus 68.7% for the frozen-policy baseline and 70.3% for the fixed schedule; over trials 31–100 the rates are 72.9% versus 66.4% and 68.7%. ASPIRE spends its 100-execution budget exploring rather than attempting the task, so it has no comparable deployment success.

• Real-world results reproduce every controller outcome. On the 60-second task, training is requested at trial 30, the candidate is adopted, and held-out success doubles from 35% to 70%. On the 40-second task, training is requested at trial 30 but the candidate is reverted; learning stays at 44% versus 46% frozen and held-out at 40% versus 40%. On the 90-second task the controller waits, and both systems reach 40% held-out. Mean held-out success rises from 38.3% to 50.0%; learning-phase means are 38.0% (Robo-COP) versus 36.7% (frozen).

• Forcing a rejected update hurts. A separate diagnostic that forced training on the 90-second task and deployed the checkpoint without verification reached only 20% held-out, half of either system.

Methodology in Plain English

Robo-COP is an embodied coding agent built on the CaP-X execution tools, extended with an execute_vla tool. Its orchestrator writes a program that calls scripted motion primitives, perception functions, and the VLA policy, conditioned on the task goal, current observation, episode history, and a task memory. A VLM monitors execution and can interrupt a call or end an episode; when execution stops, a visual judge either ends the episode or asks for revised code. The system credits VLA and scripted actions separately, so a successful skill keeps its credit even if a later step fails, and at the end of each episode the orchestrator records what worked and what failed in memory.

Two pieces make policy learning possible. The first is action unification: because the orchestrator may alternate between scripted primitives and VLA calls within one execution (for example, approaching a block with scripted motion and then calling the VLA to lower and close the gripper), the system records commanded joint targets, gripper commands, observations, and robot states in execution order. This yields one continuous action stream in the VLA's training format, so a segment can become a demonstration even though part of it was produced by scripted primitives.

The second is skill-level curation. A text judge reads the executed code, console output, and episode feedback to identify each segment's intended skill and assess its outcome. For scripted turns it groups consecutive primitive calls into motion-level skills, and a skill labeled as a pick must also include the lift. A video judge then samples frames and checks them against the proposed instruction, accepting only success or clear partial progress. Segments are kept independently of episode outcome, so a successful pickup survives even if the block is later dropped. Accepted segments are split wherever the instruction changes, a rejected row intervenes, or the joint state jumps by more than 0.3 rad; segments shorter than two steps are discarded, and each remaining segment is padded with fifteen held frames so the policy learns to stop at the end of a skill.

A language-model controller decides when to train, reviewing curated demonstrations alongside recent performance, memory, failures, and earlier attempts, and returning train or wait. Guards prevent decisions while a candidate is under evaluation, require at least thirty episodes containing a kept segment and at least ten such episodes since the last attempt, cap training at three attempts per task, and block training when the incumbent's success reference has fewer than five trials or the budget is spent. A train decision names up to four target skill instructions that guide verification; training itself uses all curated data so far, fine-tuning a candidate from the pretrained checkpoint with LoRA and the policy's original flow-matching objective. Deployment pauses while the candidate trains, and the incumbent stays in service as the baseline.

Candidate verification runs during a window of up to 25 episodes, with the deployed policy retained as a fallback. Because fallbacks can complete a task after the candidate fails, verification is performed at the skill level rather than only at the episode level. The candidate must keep task success within a margin of 0.1 of the incumbent's recent reference and must not fall more than 0.1 below the incumbent's per-call finish rate on admitted skills with enough calls. A round with too few candidate calls returns starved and the candidate is discarded with memory unchanged; a failed inequality returns revert and the incumbent stays deployed. After adoption or reversion, memory entries that discourage using the VLA are marked unverified for the current checkpoint (not deleted) and references to the candidate are rewritten to name the policy in service.

Evaluation uses ten RoboLab tasks with randomized object configurations, chosen before running Robo-COP to span three regimes: tasks the pretrained VLA solves alone, tasks scripted primitives solve, and tasks needing both. The initial policy is π0.5 trained on DROID and served at its training control rate, with Gemini 3.8 Flash running the orchestrator. Baselines are π0.5 only, a frozen-policy harness with the same episode reflection, a fixed schedule that trains every 30 trials and deploys every checkpoint without verification, and ASPIRE adapted to the benchmark's API setting. Each system runs 100 deployment trials per task on matched initializations, sharing the first 30 trials before any training decision; policies and memory are then frozen and held-out success is measured on 50 held-out initializations per task.

Why This Matters

The paper identifies a structural ceiling in agentic robot systems: an orchestrator wrapped around a frozen policy can route around failures but cannot remove them. It shows that treating each fine-tuned checkpoint as a candidate to be verified, rather than an update to be deployed, is what makes deployment-time policy learning safe enough to be worth doing at all — and that revising the orchestrator's memory after each policy change is necessary, because advice such as "use scripted motion for grasping" becomes stale the moment the policy improves.

Real-world applications:

• Long-running warehouse or fulfillment robots that must keep improving on picking and placing as object mixes drift, without a separate data-collection phase.
• Household or service robots deployed in a specific home, where each failed attempt at a task still contains usable fragments of correct behavior.
• Manufacturing or lab automation cells where a limited number of task attempts per day makes it important not to waste trials on updates that would regress performance.
• Field robotics with intermittent connectivity, where the improvement loop must run on the robot from its own executions rather than from a curated human dataset.

Industry relevance: the result that fixed-schedule fine-tuning recovers only 1.0 of the 9.0 available points, and that forcing a rejected update halves held-out success in one real-world diagnostic, is a direct argument for gating policy deployment behind verification in commercial robot fleets, where an unverified regression costs physical throughput.

Future Directions

• Improving the rest of the system. Robo-COP only updates the policy and the orchestrator's memory; the scripted skills, the judges, and the orchestrator's own reasoning stay fixed. A fully self-improving system would require every component to improve.

• Structured skill exploration inside the deployment loop. ASPIRE-style exploration, which diagnoses failures and retains validated programs as reusable skills, could run directed at the task at hand, giving the orchestrator better tools whose successful executions then become policy training data.

• Aligned orchestrator learning. The orchestrator itself could be better aligned with reinforcement learning from deployment outcomes.

• Cross-task generalization. Because curation is at the skill level, training a single policy on skills pooled from many tasks is a natural path toward a system that improves at every task it encounters the longer it is deployed.

• Scalability. The paper states that the scalability of Robo-COP still needs to be validated on larger experiments. Robo-COP can also only learn skills its policy and scripted tools already demonstrate at least partially, physical deployment requires human resets, and each update adds curation, training, and evaluation cost.

Target Audience

Researchers and engineers working on robot foundation models, VLA policy adaptation, and agentic robot systems that combine language-model planning with learned low-level control. It is also relevant to practitioners deploying robots in the field who need to decide when a fine-tuned checkpoint is safe to ship, and to readers interested in self-improving systems that learn from their own execution traces rather than from curated datasets.

Authors’ abstract

Vision-language-action (VLA) policies trained on large datasets are capable within their training domains, yet they still fail to generalize to the variety of situations a robot meets in real-world deployment. Agentic robot systems complement the policy with a vision-language model (VLM) orchestrator that learns when to call the policy, how to instruct it, and when to use scripted skills instead. However, because the harness is built around a frozen policy that has limited language steerability, the orchestrator can avoid the policy's failures but never overcome them. The policy becomes the bottleneck of the whole system. Fine-tuning the policy can remove this bottleneck, but updating it alone decouples it from an orchestrator tuned to its old behavior. We propose Robo-COP, in which the orchestrator and policy co-evolve during deployment. Robo-COP curates skill demonstrations from its own executions, fine-tunes the policy when this data can address recurring failures, and adopts each new policy only after it improves the skills it was trained for. Across ten simulated RoboLab tasks, Robo-COP raises mean held-out success from 64.8% to 73.8% over the same harness with a frozen policy, while fine-tuning on a fixed schedule without verification reaches only 65.8%. On three real-world tasks, Robo-COP raises held-out success from 38.3% to 50.0%. Robo-COP turns deployment into a self-improving flywheel in which robots learn by doing, with each improvement in execution producing better data for the next round of learning. Videos and code are available at https://robo-cop.pages.dev/.