From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation

AI summary

Overview

Research area: Robotics, specifically vision-language-action (VLA) models, hierarchical robotic planning, and agentic/self-improving systems for long-horizon manipulation.

Technical level: Intermediate. The paper assumes familiarity with VLA architectures, memory-augmented policies, and hierarchical planner–executor designs, but its central idea (an editable "harness" around a language-model planner) is explained without deep math.

Scope: The paper proposes SERA, a system that wraps a fixed high-level vision-language model (VLM) planner in an editable harness to manage task context and to revise itself from execution experience, and evaluates it on the history-dependent RoboMME benchmark (16 tasks, four suites, 50 episodes per task, 800 episodes total).

What This Paper Is About

Many manipulation tasks cannot be solved from the current camera image alone: the right action may depend on how many repetitions have already succeeded, on an object seen earlier, or on the order of locations visited in a demonstration. Standard VLA policies and even memory-augmented ones can lose track of this information, and errors compound over long horizons.

The paper's goal is to make the high-level planner in a hierarchical robot system behave more like an agent that maintains an explicit running record of task progress and execution outcomes, and then to let that record-keeping process improve itself automatically using rollouts in the environment, without collecting new demonstration data or fine-tuning the planner.

Key Contributions

1. An agentic planning framework built on an editable harness. A high-level VLM is organized as an agent through a harness configuration that specifies how inputs are assembled, how task context is maintained, how commands are checked, and how execution feedback feeds later decisions. The low-level VLA executor remains fixed.

2. Execution-driven self-evolution. Analysis and revision agents diagnose failures from execution traces and propose structured candidate changes to three harness components: planning guidance, assessment criteria, and control rules. A verifier evaluates candidates through environment rollouts. No supervised fine-tuning dataset is constructed for the high-level planner.

3. A selection-and-freeze protocol. The chosen harness configuration is fixed before held-out testing, while task context continues to update within each episode. Test outcomes do not enter the revision process, and selected candidates remain in the aggregate even if their test score decreases.

4. An empirical study on RoboMME. SERA reaches 51.38% overall success with autonomous selection, versus 44.38% without self-evolution, a gain of 7.00 percentage points, with execution-case analyses of demonstration understanding (VideoPlaceOrder) and self-evolution repair (PatternLock).

Main Findings

• Self-evolution improves overall success by 7.00 percentage points. SERA moves from 44.38% to 51.38% overall success on RoboMME after harness revision, an aggregate effect measured with the harness fixed before testing.

• SERA leads the listed methods overall (51.38%). The next-best listed method is FrameSamp+Modul at 44.51%, followed by MemER at 42.38%. For reference, the VLA baselines score 17.93% (π0.5) and 19.40% (π0.5 with action history), and SAM2Act scores 21.37%.

• Strength is concentrated in Reference and Permanence. SERA scores 69.50 on Reference and 57.00 on Permanence, the highest among listed methods on both. It scores 49.00 on Counting and 30.00 on Imitation, where FrameSamp+Modul is higher at 65.22 and 51.39 respectively.

• Per-task gains are uneven. In the paper's local reproduction comparison (50 episodes per task), SERA succeeds on 411/800 episodes versus 363/800 for the reproduced MemER. SERA is markedly higher on VideoPlaceButton (49/50 vs 14/50), VideoPlaceOrder (46/50 vs 18/50), VideoUnmaskSwap (46/50 vs 18/50), and PatternLock (30/50 vs 10/50), but lower on ButtonUnmask (20/50 vs 41/50), ButtonUnmaskSwap (0/50 vs 12/50), MoveCube (30/50 vs 45/50), and BinFill (24/50 vs 30/50). SwingXtimes ties at 36/50.

• Repairs target concrete, diagnosable failures. Revisions addressed inconsistent spatial interpretation, insufficient checks before advancing task stages, and missing action candidates, improving tasks such as SwingXtimes and PatternLock.

• Verifier feedback refines incomplete repairs. On the six-episode PatternLock revision panel, the initial harness succeeds in 2 episodes; a first revision targeting direction interpretation succeeds in 3, and inspection of a remaining failure reveals a required direction missing from the retrieved candidates; a second revision exposing all eight existing directional commands raises success to 5 of 6. The paper notes these are pre-test revision results, not an isolated comparison of the two repair components.

• Demonstration-based context can persist without re-supplying the demo. In a VideoPlaceOrder episode, the target inferred from the demonstration stays available through initialization guidance while task context records whether grasping has succeeded, so planning combines a persistent goal with changing execution state.

• Execution-level intervention remains open. In an exploratory RoboMemArena task requiring two pours before placing the container in a rack, the planner requests the second pour but the executor places the container in the rack; regrasp attempts then fail. Exploratory gripper interventions help in some development runs and fail in another.

Methodology in Plain English

The system is split into two layers. The bottom layer is an off-the-shelf VLA model that turns observations and a short command into robot actions; its weights never change. The top layer is a VLM that decides what command to issue next, wrapped in a "harness" — a structured set of prompts, context fields, and runtime rules that determine what the planner sees and how its answer is checked.

At the start of each episode, the harness builds fixed guidance from the instruction and any initial demonstration. Each planning round, it hands the planner a short written summary of task progress (bounded to 1,600 characters), relevant facts, execution outcomes, and unresolved uncertainty, along with recent images, the previous command, and a short list of allowed commands retrieved from a catalog built from executor training labels. The planner assesses what the previous command accomplished, writes an updated summary, and picks the next command in a single call. The harness validates the command's template, grounding-point count, coordinate bounds (normalized to [0, 999]) and frame validity before passing it down. Counters track repeated lack of progress (statuses include progressing, completed, uncertain, stagnant, and regressed) and can trigger reconsideration or a fresh command selection.

Self-evolution works outside the episodes. An analysis agent reads recorded observations, context, commands, and outcomes, and pinpoints where execution first diverged from intent, separating physical execution problems from task-progress reasoning problems. A revision agent turns that diagnosis into a structured candidate change with activation conditions, expressed as prompt guidance, criterion overlays, or registered control rules — the runtime rejects unregistered Python plugins, so this is not unrestricted code modification. A verifier runs the candidate on evaluation episodes and compares against cached baseline outcomes, recording repaired failures, regressed successes, and unchanged outcomes. Candidates that pass can be tested on additional episodes excluded from their diagnosis; fallbacks with one additional successful episode and no observed regression can be retained after the search budget is exhausted. Tasks with no selected revision keep the base harness.

Why This Matters

Impact on research. The paper reframes a chunk of the hierarchical-planning problem as harness engineering rather than model training: capability gains come from better context organization and procedural rules around a frozen planner and frozen executor. It also provides a concrete comparison point for the growing line of harness- and experience-adaptation work in embodied settings, and its selection protocol — freezing the harness before held-out testing, keeping degraded selected candidates in the aggregate — is a methodological detail other self-improving robot papers will have to match.

Real-world applications:

• Household and service robots doing multi-step chores where the right move depends on what already happened (how many items were placed, which cabinet was opened first).
• Warehouse pick-and-place and kitting, where a task must be repeated a specified number of times and progress must be tracked across grasps and retries.
• Instruction-following from human demonstrations, such as "put it where the second object was," which requires retaining a target seen earlier even after the scene changes.
• Long unattended deployments, where a robot must recover from its own execution errors — a dropped object, a mis-grasp, a skipped stage — without a human resetting the task.

Industry relevance. The approach lets teams improve a deployed robot stack by editing prompts and control rules validated in simulation or on development episodes, rather than collecting new demonstration datasets and retraining a planner. That is a cheaper iteration loop, and it keeps the safety-relevant executor unchanged. The paper's own caveats matter commercially too: planner latency accumulates in the control loop, and the reported per-task results are mixed, so the aggregate gain is not a uniform improvement.

Future Directions

• Reducing planning latency. The paper explicitly flags VLM inference over observations and context as accumulating overhead in long-horizon execution, and leaves the trade-off between reasoning quality and responsiveness to be systematically evaluated. More efficient context processing and reasoning strategies are proposed as future work.

• Correcting execution deviations, not just replanning. The RoboMemArena case shows context can correctly retain an unfinished requirement while the executor still does the wrong thing. The authors suggest complementing task-state maintenance with mechanisms for selecting and assessing execution-level interventions, and note that reliable recovery remains an open problem given mixed exploratory gripper-intervention outcomes.

• Separating the contribution of individual repair components. The PatternLock panel shows two sequential repairs helping, but the paper states this is not an isolated comparison of direction guidance versus the expanded candidate set, leaving ablation of harness components as open work.

• Understanding when harness revision hurts. Per-task results include cases where SERA trails a reproduced baseline (for example ButtonUnmaskSwap at 0/50 versus 12/50, and MoveCube at 30/50 versus 45/50). Diagnosing why some tasks regress under the selected harness, and making selection robust to that, is a natural next step.

Target Audience

Researchers and engineers working on VLA models, hierarchical robot planning, and memory for long-horizon manipulation, particularly those interested in agentic scaffolding and experience-driven self-improvement without retraining. It also suits practitioners who want a concrete recipe for wrapping a frozen planner and frozen executor with an inspectable, revisable control layer, and benchmark-oriented readers who need the RoboMME comparison numbers and their caveats. Readers without background in VLA policies or planner–executor hierarchies will find the method readable but the baseline landscape harder to place.

Authors’ abstract

Vision-language-action (VLA) models achieve strong performance across embodied manipulation tasks by unifying visual understanding, language conditioning, and action generation. However, long-horizon tasks can involve non-Markovian dependencies: successful decisions may require retaining task progress and past interaction outcomes, while partial observability and execution errors compound over time. Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution. We organize the high-level VLM as a robotic agent through an editable harness to maintain task context and improve planning through execution experience. We instantiate this approach as SERA (Self-Evolving Robotic Agent), with a fixed VLA executor providing low-level control. The harness maintains and updates task context using observations and execution feedback to guide subsequent planning. Self-evolution uses environment rollouts to propose and evaluate harness revisions, with the selected configuration fixed before held-out testing. On RoboMME, SERA achieves 51.38% overall success with autonomous selection, compared with 44.38% without self-evolution. Our exploration suggests that agentic planning is a promising direction for history-dependent robotic manipulation.