Decoupling Planning and Control for Instructable Agents

AI summary

Overview

• Research area: Artificial Intelligence — embodied agents, reinforcement learning, vision-language models, hierarchical planning and control, multi-agent coordination.
• Technical level: Intermediate (requires some familiarity with RL, world models, and VLMs, but the core idea is conceptually simple).
• Scope: The paper introduces Instruct-to-Act, a framework that separates a high-level VLM planner from a low-latency, language-conditioned world-model controller, and evaluates it across seven embodied environments including three multi-agent ones.

What This Paper Is About

Pre-trained, instruction-tuned vision-language models (VLMs) are good at turning instructions and observations into high-level plans, but they cannot reliably produce fast, low-level action sequences in unfamiliar environments. Conversely, world-model controllers act quickly and smoothly from pixels but cannot be steered by open-ended language. This paper combines the two: a VLM planner emits sparse, high-level text instructions, and a controller trained with post-hoc language supervision executes them at control frequency.

Key Contributions

1. Instruct-to-Act, a plug-and-play planner-controller paradigm. A pre-trained VLM planner maps observations to natural-language instructions, and an environment-specific controller maps instructions plus observations to low-level actions. No VLM is fine-tuned to any environment's action space.
2. A language-aware world model and policy. The controller is an instruction-conditioned RSSM whose representation learning, actor head, and critic head are all conditioned on a latent embedding of the planner's instruction, with an added head predicting instruction completion.
3. Post-hoc instruction supervision. Segments of the controller's own on-policy rollouts are relabeled with VLM-generated instructions, and a behavior-cloning objective is added to the standard Dreamer world-model, actor, and value losses. No expert demonstrations are used.
4. Asynchronous inference and language-based multi-agent coordination. The planner reasons and plans in text space while the controller executes actions concurrently, and one shared controller architecture serves multiple agents that coordinate purely through a language chatroom.

Main Findings

• Decoupling beats direct VLM action generation. Conditioning a trained controller on sparse GPT-4o instructions improved performance on every task relative to the same VLM producing low-level actions directly. GPT-4o as planner scored 891 (Atari), 11.7 (Minecraft Diamond), 14.1 (Crafter), 76 (DMLab), 193.2 (Overcooked), 70.1 (Pico Park), and 70.2 (MindCraft), versus 670, 10.4, 8.7, 56, 180.4, 50.3, and 50.2 for GPT-4o without a controller.
• Language guidance improves over controller-only. The controller-only baseline scored 809, 8.2, 12.6, 67, 170.2, 30.7, and 40.0 across the seven tasks, lower than every planner+controller variant on the language-relevant tasks.
• Strong planners lead. GPT-4o and Qwen-VL-2.5-72B achieved the best overall results among the four planners tested (the others being Gemma-3-27B and llava-v1.6-34b).
• Competitive with specialized baselines. The method remained competitive with vision-language-action and multi-agent RL baselines on six of seven tasks, including JARVIS-1 on Minecraft (11.7 vs. 12.3 for GPT-4o as planner) and QMIX on Overcooked (193.2 vs. 187.2).
• High instruction-following accuracy. Controllers achieved an average instruction-following accuracy of 92.8%, with per-domain reliability between 172/200 and 194/200 (86% to 97%).
• Online planning outperforms offline and scales better. On Minecraft Diamond, online planning improved task performance over offline planning as model sizes increased and remained an order of magnitude faster in throughput across small (50M controller with 2B planner), medium (200M controller with 7B planner), and large (800M controller with 72B planner) configurations.
• Decentralized multi-agent communication helps. In Pico Park, decentralized communication gave better task performance than centralized control while maintaining equivalent throughput as the number of agents varied from 2 to 8.
• Semantic content matters more than annotation quality. Environment-derived symbolic labels slightly beat VLM-generated instructions on Minecraft Diamond (11.4 vs. 11.1), but VLM annotation needs only raw observations. Template-based instructions scored 9.9, clustered action labels 8.9, random strings 8.3, and no instructions 8.2.
• Controllers generalize across planners. Train-times-eval cross-planner transfer stayed stable across all planner pairs, ranging from 9.2 to 10.2 on Minecraft Diamond.
• Architecture flexibility. A transformer world model was comparable to the RSSM (11.2 vs. 11.0), while weaker variants degraded performance (RNN policy: 9.7; value-only world model: 10.2).
• VLM-decided instruction timing beats fixed cadence. GPT-4o scored 10.75 with VLM-decided proactiveness versus 9.60 with fixed cadence.
• One controller per environment is currently the practical choice. A single controller trained jointly on all environments lost 1.12 points on Minecraft Diamond, roughly 10% relative.

Methodology in Plain English

The system splits the job of an embodied agent into two parts that run in parallel processes. The planner is an off-the-shelf instruction-tuned VLM that looks at visual observations, maintains a memory bank and a partial plan, and occasionally emits a short text instruction. The controller is a recurrent state space model (RSSM) that reads the instruction through a frozen language encoder and produces low-level actions in real time.

Training the controller does not require expert demonstrations. The controller first collects its own rollouts in the environment, guided only by reward. Then a VLM annotator (GPT-4o) looks back at random segments of those rollouts — each spanning 1 to 20 steps, with 50% of the replay buffer annotated — and writes a short high-level instruction describing what happened in each segment. Those instructions are encoded and attached to the corresponding transitions, and the controller is trained with an added behavior-cloning loss to reproduce the actions it took under that instruction. An extra head is trained with binary cross-entropy to predict when an instruction has been completed, so the planner knows when to send a new one. Unannotated portions of the buffer let the controller keep acting autonomously from reward alone, using a learned null embedding when no instruction exists. The final objective is the standard Dreamer world-model, value, and actor losses plus the behavior-cloning and stop-prediction losses.

For multi-agent settings, the same controller parameters are shared across agents, but each agent maintains its own latent state and acts on its own observation stream. Planners exchange messages through a shared chatroom, either in a decentralized mode where any planner may message a peer, or a centralized mode where a fixed hub agent broadcasts. Coordination emerges in language rather than through a centralized critic.

Why This Matters

• Impact on research: The work shows that a stable natural-language interface between a slow reasoner and a fast controller lets any pre-trained VLM be plugged in as a planner without fine-tuning to an environment's action space, offering an alternative to monolithic vision-language-action models and to classic centralized-training/decentralized-execution multi-agent RL.
• Real-world applications:
  • Robotics, where high-level task descriptions must be translated into long, low-level control sequences on physical hardware.
  • Autonomous driving, where slow deliberate reasoning must be separated from fast reactive control.
  • Long-horizon open-ended games such as Minecraft, where agents must plan, craft, and build over thousands of steps.
  • Multi-robot coordination, where cooperative tasks require agents to negotiate roles in real time.
• Industry relevance: The controller is a one-time, environment-specific cost of roughly 23 GPU-hours on 4 RTX A6000 GPUs (about 5.5M environment steps per environment), while the expensive VLM stays plug-and-play. Post-hoc annotation adds roughly 17% GPU-hours and a 12% training slowdown. This makes the approach cheaper to adapt than fine-tuning a large VLA model per domain.

Future Directions

• Joint multi-task controllers. Early experiments showed a single controller trained on all environments lost 1.12 points on Minecraft Diamond, so lightweight domain-general instruction-conditioned controllers remain future work.
• Human planners in the loop. The authors list studying human planners directly as an important next step, alongside a lightweight human experiment reported in the appendix.
• Asymmetric multi-agent roles. Extending the framework beyond symmetric shared-controller agents to asymmetric roles and multi-task controller training is called out as future work.
• Better planner-controller alignment. Improving alignment through prompting or lightweight planner adaptation is identified as an open direction, along with generalizing behavior-cloning supervision to domains without environment metadata.

Target Audience

Researchers and practitioners working on embodied AI, reinforcement learning, world models, and vision-language-action systems, as well as engineers who need to deploy large language models in settings where abstract guidance must become long streams of low-level control. Readers with some background in RL or multimodal models will get the most from the experimental sections; the core idea of splitting planning from control is accessible to a broader technical audience.

Authors’ abstract

Recent work shows that pre-trained, instruction-tuned vision-language models (VLMs) perform well at mapping from instructions and observations to high-level plans, but struggle to realize such plans as reliable low-latency action sequences in unfamiliar environments. At the same time, world-model controllers excel at fast observation-to-action control, but lack open-ended task guidance. In this work, we combine these strengths into a single system, Instruct-to-Act, where we train a world-model controller to act autonomously at high frequency when conditioned on sparse, higher-latency, and high-level text instructions generated by a VLM planner. To train controllers to be language-instructable, we relabel segments of controller policy rollouts with synthetic instructions and jointly optimize a behavior-cloning objective along with existing reward-maximizing and world-modeling objectives. We evaluate our proposed approach across seven embodied environments, including three multi-agent environments where VLM planners coordinate through language while trained controllers serve as their actuators. Under matched observation and action spaces, our decoupled approach consistently outperforms controller-only and direct VLM action-generation variants, preserves fast control, and lets us swap in different pretrained VLM planners without fine-tuning, while remaining competitive with strong vision-language-action and multi-agent RL baselines on six of seven tasks.