EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents

AI summary

Overview

Research area: Robotics — vision-language-action (VLA) models and embodied agents for long-horizon robot manipulation.

Technical level: Advanced. The paper assumes familiarity with VLA policies, action chunking, hierarchical control, and reinforcement learning terminology, though the core idea is explained clearly enough for a determined non-specialist.

Scope: The paper proposes a framework for wrapping existing low-level VLA policies in a guarded, closed-loop agent runtime with explicit skills, preflight validation, post-action verification, and recovery.

What This Paper Is About

A VLA model predicts actions from images and instructions, but a long-horizon robot task needs more than action prediction: something has to decide what subgoal to attempt, check whether that subgoal is actually executable in the current physical state, confirm afterward that it worked, and recover if it did not. Existing VLA policies leave these intermediate decisions implicit (making failures hard to diagnose), while LLM-style robot agents make decisions explicit but do not guarantee a proposed skill is valid or executable in the current state, since proposals can rely on stale observations, omit required arguments, or fail silently.

The paper's goal is to build a middle layer — an "agentic layer" — that bridges learned decision making and physical execution so that proposed operations are validated before execution, verified afterward, and logged as structured evidence for both diagnosis and training.

Key Contributions

1. A skill-oriented closed-loop AgentLoop. Long-horizon VLA agents are formulated as closed-loop systems coordinating explicit embodied skills across observation, planning, readiness checks, bounded execution, progress verification, and recovery — rather than as a fixed, one-pass pipeline.

2. Policy–runtime separation. A shared skill contract lets the policy propose structured skill decisions while a separate runtime enforces prerequisites, artifact freshness, action validity, and legal state transitions, making failures explicit and agent trajectories diagnosable.

3. Modular adaptation and cross-benchmark validation. Independently trainable and replaceable interfaces are defined for planning, verification, action generation, and environment interaction, with the resulting low-level VLA policy instantiations evaluated on RoboTwin 2.0 and LIBERO.

4. Structured trajectory logging for component-level training. The same interface records planning, execution, verification, and recovery events as structured trajectories, supporting supervised adaptation of individual components (planner, verifier, scheduler, low-level policy) and optional online optimization (group-relative policy optimization) when interactive feedback and a reliable evaluator are available.

Main Findings

• RoboTwin 2.0 execution performance: Task-adapted low-level VLA policies achieve 86.20% macro-average success across 50 RoboTwin 2.0 tasks, compared with 82.74% for the π0.5 reference reported by LingBot-VA — an improvement of 3.46 percentage points.

• Largest per-task gains: The biggest improvements over the reference occur on Hanging Mug (+20), Blocks Ranking Size (+15), Open Microwave (+15), Move Can Pot (+10), and Move Stapler Pad (+10).

• Baseline comparison on RoboTwin 2.0 (averages): ACT 29.7%, DP 28.0%, RDT 34.5%, DP3 55.2%, π0 65.9%, X-VLA 72.8%, π0.5 82.74%, and the paper's task-adapted policies 86.20%.

• LIBERO execution performance: Across LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, the VLA policy instantiation achieves 97.40% average success, compared with 96.85% for the official OpenPI reference.

• Memory-dependent tasks: On four memory-dependent RMBench tasks, the same task-adapted execution approach reaches 12.5% average success — a markedly lower figure, presented as an additional evaluation of subtask-conditioned execution when the correct action depends on prior interaction history.

• Controlled loop ablation: Three controlled AgentLoop ablations were run on all 50 RoboTwin 2.0 tasks with 100 episodes per task, sharing the same planner, low-level policy, initial states, and terminal evaluator, differing only in whether semantic subtasks, intermediate verification, and repeated action chunks are available. The numeric ablation results are not reported in the provided paper content.

• Instantiation choices: The framework is instantiated with Qwen3-VL-based agent components and an OpenPI/π0.5 low-level VLA policy. The high-level scheduler is supervised on deployment-consistent decision traces while the low-level VLA policy is kept frozen, isolating the SFT stage from continuous-control learning.

Methodology in Plain English

The system is structured as a guarded finite-stage controller over "executable embodied skills." At each loop step, the agent maintains a state consisting of the current phase, a set of task artifacts (observations, plan, active subgoal, execution report, verification report, recovery context), and an ordered history of prior decisions.

A high-level vision-language model policy receives a compact context — the instruction, current phase, artifact summaries, recent errors, and a bounded summary of recent decisions — and picks one of three control types: RunSkill(k, q), AdvanceStage, or FinishRun. The set of usable skills is filtered to those compatible with the current state.

Each skill is defined by a contract with typed inputs and outputs, explicit prerequisites, an executable operation, a resulting state update, and a failure mapping. Proposed operations are treated as proposals until their schema and prerequisites pass validation; artifacts carry provenance and freshness markers so stale predictions cannot authorize new physical actions.

The loop runs through six phases — Observe, Plan, Preflight, Execute, Verify, Recover — but these are not a rigid pipeline: the policy can revisit earlier phases when new evidence invalidates the plan. Before execution, the runtime checks phase compatibility, required inputs, artifact freshness, action validity, and legal state transitions; blocked decisions are recorded with their evidence rather than silently executed.

In the Execute phase, the low-level VLA policy maps the active subgoal, current observation, robot state, and preflight evidence to a bounded action chunk, which the runtime accepts only if the horizon is within limits, the chunk matches the policy's action schema, it is fresh relative to the subgoal and observation, and the commands are numerically valid. After execution, the Verify phase judges only the active subgoal from fresh post-action evidence and selects one of six semantic routes: Advance, Continue, Reobserve, Replan, Recover, Finish. Overall task success is still defined by the environment's own evaluator, not by the local subgoal judgement.

For training, the primary mechanism is component-level supervised adaptation: the planner learns to produce executable subgoals, the low-level VLA policy is adapted with subtask-level demonstrations, and a verifier can be trained from post-execution observations and subgoal-completion labels. Training samples follow the same input boundary used at deployment, and loss is applied only to the tokens generated by the learned component, not to prompts, images, or environment messages. Optional closed-loop adaptation using group-relative policy optimization is described but positioned as a refinement mechanism rather than the source of the headline results.

Why This Matters

Impact on research. The paper reframes the question from "how good is the VLA policy?" to "how do we build a reliable system around a VLA policy?" By separating policy proposals from runtime-enforced execution, it makes failures attributable — object grounding, subgoal selection, low-level execution, verification, or recovery — which is difficult with end-to-end policies and unverified with prompt-only agents. Because the skill interface is fixed, low-level policies can be swapped or adapted without redefining the agent loop, which makes the framework a plausible substrate for comparing and composing future VLA models under a common runtime abstraction across benchmarks.

Real-world applications.

• Warehouse and logistics robots that must chain pick, place, and sort operations over long horizons, where knowing why a step failed is as important as the step itself.
• Household service robots executing multi-step tasks such as "place the container on the plate," where object configurations and partial observability change between attempts.
• Industrial bimanual manipulation on structured workcells, where preflight checks can prevent an invalid action from ever reaching the hardware.
• Deployment monitoring and post-incident diagnosis: the structured trajectory log turns terminal success or failure into an inspectable record of decisions, verifications, and recoveries.

Industry relevance. The framework is deliberately complementary to existing VLA policies rather than a replacement, so teams already invested in a low-level policy (the paper instantiates OpenPI/π0.5) can add an agentic layer without retraining the controller. It also targets a practical pain point in deployment: distinguishing infrastructure failures from agent-policy failures, which the paper records separately.

Future Directions

• Closing the memory gap. The 12.5% average success on four memory-dependent RMBench tasks versus 86.20% on RoboTwin 2.0 and 97.40% on LIBERO points to a clear open problem: making subtask-conditioned execution work when the correct action depends on prior interaction history.
• Reporting the ablation outcomes. The three controlled AgentLoop ablations (semantic subtasks, intermediate verification, repeated action chunks) were run on all 50 RoboTwin 2.0 tasks with 100 episodes each, but their numeric results are not reported in the provided content; quantifying each component's contribution is a natural next step.
• Moving online optimization from optional to central. The paper describes group-relative policy optimization with episode-level attribution that it calls "necessarily coarse" — multiple decisions in one episode receive the same return even when their causal contributions differ. Developing finer-grained credit assignment for physical, partially observed tasks is an open problem.
• Broadening the verification and recovery machinery. Verification currently judges only the active subgoal, and there is little detail on how recovery policies are trained or how often recovery succeeds; systematic study of the Recover phase, and of verification quality under contact-rich failures, would strengthen the framework.

Target Audience

Robotics and embodied-AI researchers working on VLA policies, hierarchical control, and long-horizon manipulation; engineers building deployment stacks around learned robot policies who need diagnosable failure modes and swappable controllers; and agent researchers interested in how verifier feedback and tool-like skill calls transfer from software environments to physical ones. Readers without a background in robot learning will find the framework's structure accessible, but the experimental sections assume familiarity with benchmarks such as RoboTwin 2.0, LIBERO, and RMBench, and with policies such as ACT, DP, RDT, DP3, π0, X-VLA, and π0.5.

Authors’ abstract

Vision-language-action (VLA) models map visual observations and language instructions directly to robot actions, but long-horizon tasks require more than action prediction. An agent must coordinate perception, planning, execution, progress verification, and recovery as the physical state evolves. An action prediction or a model-generated skill decision does not, by itself, guarantee that the proposed operation is valid in the current state or that its outcome will be verified. We propose EmbodiedSkills, a unified framework that treats each skill decision as an execution proposal: the runtime checks its prerequisites before execution and verifies the outcome afterward. A shared executable-skill interface connects high-level skill selection, bounded low-level VLA execution, and post-action verification within a single agent loop. Because this interface remains fixed, low-level VLA policies can be replaced or adapted without changing the agent loop. The interface also records planning, execution, verification, and recovery events as structured trajectories, which provide supervision for individual components and can support optional online adaptation when interactive feedback is available. We instantiate EmbodiedSkills with Qwen3-VL and OpenPI/pi0.5 on RoboTwin 2.0 and LIBERO. Task-adapted low-level VLA policies achieve an average success rate of 86.20% across 50 RoboTwin 2.0 tasks and 97.40% across the four LIBERO suites. These results establish the execution performance of the task-adapted low-level VLA policies used in EmbodiedSkills. On four memory-dependent RMBench tasks, the same task-adapted execution approach achieves 12.5% average success. The framework provides a trainable and inspectable agent layer for turning these policies into closed-loop embodied systems.