WorldGuide: Goal-Directed Video World Model for Procedural Task Execution

AI summary

Overview

Research area: Computer Vision — video generation and video-based world models, specifically closed-loop procedural task execution.

Technical level: Advanced. The paper combines a vision-language planner, a diffusion-transformer video executor, and a hierarchical memory-compression scheme, and reports results against seventeen video-generation baselines.

Scope: WorldGuide reformulates procedural video generation as closed-loop task execution in visual world space, where a planner selects each atomic action from the visual state it just generated and decides when the task is complete.

What This Paper Is About

Video generators can produce visually plausible clips, but they cannot follow a multi-step procedure toward a goal: open-loop systems fix the prompt and the number of steps before rollout, so a wrong or incomplete step cannot be corrected and generation may continue after the task is finished. Existing closed-loop systems narrow this gap but still pair a planner with a frozen or only indirectly supervised executor, so a correct plan can still fail to be realized on screen. WorldGuide closes this gap by training both a planner and an executor on the same step-level procedural demonstrations, so the planner predicts the next atomic action or a completion token from the generated visual state, and the executor is directly trained to render exactly those actions.

Key Contributions

1. A closed-loop formulation of procedural video generation. WorldGuide treats goal-directed video generation as closed-loop task execution in visual world space: given only an initial image and a task goal, it predicts an atomic action, generates the corresponding clip, and uses the generated result to choose the next action or to terminate.

2. Jointly supervised Planner and Executor trained on shared demonstrations. The ContextPlanner (initialized from Qwen2.5-VL-7B) is trained to predict the next atomic action or the completion token <|Task Completed|> from visual progress, and the Executor (initialized from HunyuanVideo-1.5) is conditioned on the frozen ContextPlanner's action embeddings so it learns to realize the same atomic actions the planner proposes. Planning and execution are kept separate by a natural-language interface.

3. WorldGuide Bench, a step-level procedural dataset. Approximately 59K step-annotated videos (58,679) across 245 tasks and 27 procedural categories, with temporally grounded atomic action clips, task goals, and explicit completion signals. The paper reports this is the only dataset in its comparison table that provides step-clip annotations, atomic-action granularity, task goals, and completion supervision together.

4. Hierarchical compressed visual memory with a bounded token cost. The Executor maintains memory that represents older history at progressively coarser spatial resolution, with a token budget bound of ≤ 8 < 9 full-resolution latent-frame equivalents and total history length F ≤ T + 1 ≤ 1366.

Main Findings

• Closed-loop execution improves task completion. WorldGuide raises Task Success from 11.71% (open-loop variant) to 33.33% and Plan Accuracy from 22.87 to 62.85 on WorldGuide Bench, a 21.62% Task Success improvement over the open-loop variant.

• WorldGuide beats a larger model that receives reference plans. On WorldGuide Bench it reaches 33.33% Task Success versus 29.90% for MiniMax-H3-33B, even though MiniMax-H3 receives reference action plans. On Video-CraftBench under goal-only conditioning it reaches 47.69% versus 32.73% for MiniMax-H3.

• Visual feedback is necessary for replanning. Removing generated clips from the planner's input (keeping goal and previous actions, with the Executor unchanged) drops Task Success from 33.33% to 14.72% and Plan Accuracy from 62.85 to 29.29 — an 18.61% Task Success gap.

• Memory improves execution and downstream planning. With checkpoints unchanged, enabling memory improves Task Success by 5.77% on WorldGuide Bench and 17.91% on Video-CraftBench, and improves video-judged Plan Accuracy by 7.49%.

• The planner can recognize completion. In autonomous rollout, using generated visual feedback improves Plan Success from 86.41% to 91.75% and reduces Repeat/Skip from 20.19 to 15.73. With ground-truth histories, the planner reaches 79.66 Plan Accuracy at K=3, versus 64.03 in autonomous rollout.

• Planning is the tighter bottleneck. Replacing the learned planner with oracle reference actions raises Task Success by +5.49% using the same trained Executor and memory.

• Procedural gains do not uniformly transfer to visual quality. WorldGuide achieves the highest Aesthetic Quality (49.36) and Overall Consistency (31.13) on WorldGuide Bench, but on Video-CraftBench it shows modest drops relative to MiniMax-H3 in Imaging Quality (64.63 vs. 69.65) and Overall Consistency (22.09 vs. 23.18).

• Backbone choice matters. HunyuanVideo-1.5 has 8.3B parameters, fewer than many evaluated generators, yet achieves the highest Plan Accuracy (43.53) and Task Success (12.00%) of all baselines except the 33B MiniMax-H3 on WorldGuide Bench under reference actions.

• Failures are coupled. Failures typically arise from a premature or redundant planner action, or an Executor clip that does not fully realize the intended state transition; because each generated state feeds the next decision, such errors can propagate.

Methodology in Plain English

The loop. WorldGuide starts from an initial image and a task goal. At each step, a ContextPlanner looks at the goal, the current visual state, and the last K = 3 clip–action pairs, and outputs either a single natural-language atomic action or a completion token. If it outputs an action, the Executor generates a short video clip showing that action, using the clip's last frame as the reference for the next step. That new clip becomes the next visual state, is paired with its action in the history, and updates the memory. The rollout stops when the planner predicts completion or the step budget is reached. All generated clips are concatenated into the final video.

Two models, trained in sequence. The planner is a supervised fine-tune of Qwen2.5-VL-7B-Instruct on demonstration histories and their next-action or completion targets, using token-level cross-entropy masked to response tokens — no architectural changes or auxiliary objectives. It is then frozen, and the Executor (a HunyuanVideo-1.5 diffusion transformer) is fine-tuned with a flow-matching objective to predict the residual velocity toward the target, conditioned on the frozen planner's action embeddings plus visual context. Freezing the planner is deliberate: if the flow-matching loss also updated it, its parameters would be optimized for embeddings that ease video generation, and its language outputs could drift away from faithful instructions.

A language-level interface. Each predicted action is a natural-language instruction, which the paper argues makes guidance interpretable, failures diagnosable (predicted actions can be swapped for reference actions while keeping the Executor fixed), and plans independently evaluable as text.

Bounded memory. YUME's spatial history-compression schedule is adapted: recent latent frames get finer spatial embeddings than older ones, using multi-scale 3-D convolutions with kernel and stride (1, pr, pr) for r in {1, 2, 4, 8, 16} and spatial patch size p = 2. The coarsest embedder is E₆₄ = E₁₆ ∘ Φ, with effective spatial stride 128 = 64p. The oldest history latent frame is kept as an anchor. The resulting token budget stays bounded at ≤ 8 < 9 full-resolution latent-frame equivalents.

Data. Because step-level action–video supervision did not exist for joint planner–executor training, the authors built WorldGuide Bench using Gemini 2.5 Flash to filter web instructional videos and annotate temporally grounded atomic actions and completion. Evaluation used 980 test samples from WorldGuide Bench and all 294 Video-CraftBench samples.

Evaluation. Visual quality and video–text consistency use VBench, with Paired Alignment for generated-versus-reference similarity. Task execution is judged by Gemini: Task Success Rate, Plan Accuracy, Order Score, and Repeat/Skip. GPT-5.2 evaluates predicted action sequences for Plan Success. A human study is also reported.

Why This Matters

Impact on research. The paper argues that visual plausibility does not imply task completion, and that the gap between deciding an action and realizing it is the core obstacle for long-horizon procedural generation. By training the planner and executor on the same demonstrations and letting generated outcomes drive both the next action and the stopping decision, it offers an alternative to open-loop synthesis and to closed-loop systems that pair a planner with a frozen executor or an indirect critic. It also supplies step-level action–video supervision that the paper reports was previously unavailable.

Real-world applications (grounded in the paper's own task categories):

• Cooking guidance, such as the step-by-step fried rice tutorial shown in the paper's qualitative comparison.
• Origami and other craft procedures where a mistake in one step propagates to everything after it.
• Assembly tasks, where prerequisite ordering is essential, which the Order Score metric directly targets.
• Generating instructional videos from web demonstrations, the source material for WorldGuide Bench.

Industry relevance. The result that a smaller 8.3B-parameter system can beat a 33B model that receives reference action plans is directly relevant to anyone building procedural assistants, robotics-adjacent world models, or automated instructional-video pipelines. The paper keeps the full system at the size of the original HunyuanVideo-1.5 pipeline by reusing the planner as the Executor's text encoder, which matters for deployment cost. The bounded memory design targets the practical constraint of inference cost growing with rollout length.

Future Directions

• Joint optimization of planner and executor. The paper names this as the natural next step, noting the Executor already conditions on the ContextPlanner's action embeddings and that the current version keeps the language interface fixed during Executor training.

• Richer state-validity signals and explicit recovery. The authors propose strengthening the coupling so the system can detect and recover when a step is incomplete or inconsistent, rather than letting the error propagate into later decisions.

• More selective memory retention. Bounded visual memory compresses older states to control inference cost; the paper suggests that retaining task-relevant information more selectively could improve long-horizon consistency while preserving that efficiency.

• Better balance between procedural correctness and visual fidelity. WorldGuide prioritizes task execution over visual fidelity in challenging cases, with modest drops in some visual-quality metrics on Video-CraftBench; the paper calls improving this balance an important direction. Remaining execution and stopping errors also persist even in the best configuration.

Target Audience

Researchers and engineers working on video generation, video-based world models, and planner–executor systems; practitioners building step-by-step procedural assistants for cooking, assembly, or craft tasks; and anyone who needs a dataset of step-annotated instructional videos with task goals and completion signals. Readers should be comfortable with diffusion transformers, flow matching, and vision-language models, though the paper's central argument — that a procedure should unfold through decisions grounded in what was actually generated — is stated in accessible terms.

Authors’ abstract

Video generators and video-based world models can synthesize plausible visual trajectories, but long-horizon procedural tasks require generation to adapt to what has actually been produced. A model must determine the next action from its generated state, execute that action, and recognize when the task is complete. Open-loop generation cannot adapt to execution outcomes, while existing closed-loop systems often rely on pretrained executors or indirect verification. This leaves a gap between deciding an action and successfully realizing it. We formulate procedural video generation as \emph{closed-loop task execution in visual world space} and introduce \textbf{WorldGuide}. Given only an initial image and a task goal, WorldGuide predicts an atomic action, generates its corresponding video clip, and uses the generated result to select the next action or terminate. The Planner and Executor are trained on the same step-level procedural demonstrations: the Planner learns to predict the next atomic action or task completion from visual progress, while the Executor is directly trained to realize the predicted actions. Hierarchical visual memory maintains state across long-horizon execution with bounded history token cost. Due to the lack of step-level action-video supervision for joint planner-executor training, we introduce \textbf{WorldGuide Bench}: approximately 59K step-annotated videos across 245 tasks and 27 procedural categories. WorldGuide achieves a 33.33\% Task Success on \textbf{WorldGuide-Bench}, compared with 29.90\% for the strong recent video model MiniMax-H3, even though MiniMax-H3 receives reference action plans, and achieves 47.69\% on \textbf{VideoCraft-Bench} compared with 32.73\% for MiniMax-H3 under goal-only conditioning. These results demonstrate the importance of coupling planning with learned execution for goal-directed procedural video generation.