Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

AI summary

Overview

• Research area: Robotics — robot manipulation policy learning, specifically zero-shot cross-task generalization via in-context learning from human video prompts.
• Technical level: Advanced. The paper builds on flow matching, causal video-action Transformers, Mixture-of-Transformers architectures, and RoPE positional offsets; the data pipeline also involves VLMs, image editing models, and video generation models.
• Scope: One sentence — Zero-WAM is a causal video-action model that executes manipulation tasks it has never been trained on by following a human demonstration video supplied at deployment time as an in-context prompt.

What This Paper Is About

Robot policies usually rely on language to specify what task to perform, but language underspecifies manipulation: spatial constraints, intermediate states, and temporal structure are hard to articulate, and text gives no visual evidence of how the scene should evolve. The authors argue that the natural task specification for manipulation is a human video, since it directly shows the desired visual state changes over time. The goal is to train a single policy that takes such a video as an in-context prompt and executes the corresponding unseen task on a robot, without any parameter updates at deployment.

Key Contributions

1. A formulation of zero-shot robotic task generalization as in-context world-action modeling, where one causal policy accepts either a language instruction or a human video as the task specification and autoregressively predicts future robot videos plus aligned executable actions.
2. An automatic in-context human video generation pipeline that converts task-sampled robot trajectories into semantically matched human manipulation videos, producing the HumanGen dataset of 74.2K human-robot in-context learning (ICL) pairs across 8.6K tasks. The same task-level sampling yields Task-diverse VA, a task-balanced corpus of more than 6,000 tasks and approximately 400K robot trajectories per training epoch.
3. An in-context future chunk prediction (IFP) objective that supervises multiple strided future robot video chunks from the current robot-video representation, discouraging the policy from ignoring the video prompt and relying on shortcut cues from robot history and text.
4. Demonstration of zero-shot cross-task generalization on seven unseen RoboTwin 2.0 tasks and on real-world task families involving multi-object scenes, long-horizon manipulation, and fine-grained insertion, without collecting corresponding robot data or updating parameters.

Main Findings

• Simulation average success rate: On seven unseen RoboTwin 2.0 tasks, the abstract reports Zero-WAM achieving a 47.0% average success rate, described as an absolute improvement of 29.5 percentage points over the strongest video-action baseline. The introduction states the average as 46.95% and the margin over LingBot-VA as 29.50 percentage points.
• Per-task comparison (Table 2, as far as the provided content shows):
  • Place object on scale: Zero-WAM 24.67 ± 2.05, LingBot-VA 6.17 ± 4.87, WAN-Action 3.00 ± 2.16.
  • Stamp seal: Zero-WAM 47.00 ± 4.55, WAN-Action 7.33 ± 1.25, LingBot-VA 3.67 ± 2.49.
  • Open microwave: Zero-WAM 59.00 ± 2.83, LingBot-VA 29.33 ± 10.66, WAN-Action 2.26 ± 1.60.
  • Move stapler to pad: Zero-WAM 69.14 ± 2.93, LingBot-VA 23.33 ± 8.22, WAN-Action 10.67 ± 1.70.
  • Place bread in basket: Zero-WAM 35.00 ± 3.74, LingBot-VA 17.33 ± 6.18, WAN-Action 15.26 ± 2.55.
  • The remaining task rows (place empty cup, stack blocks three) and the macro-average row are not visible in the provided truncated content.
• Unseen task categories exercised: The seven evaluation tasks are place object on scale, stamp seal, open microwave, move stapler to pad, place bread in basket, place empty cup, and stack blocks three — covering pick-and-place with unseen objects and containers, unseen articulated-object manipulation, and unseen long-horizon manipulation.
• Real-world generalization: Real-world experiments report that Zero-WAM follows human video guidance to generalize to unseen task configurations and outperforms LingBot-VA across all three real-world task families (multi-object scenes, long-horizon manipulation, precision-demand insertion). Specific success figures for these real-world experiments are not reported in the provided content.
• Dataset scale compared with prior work: Table 1 lists prior task-level paired human-robot datasets — MIME (8.3K samples, 20 tasks), EgoMimic (2.1K, 3), BC-Z (18.7K, 100), EgoHumanoid (1.2K, 4), RH20T (110K, 147), EgoScale (10.3K, 344) — all manually collected. HumanGen reports 74.2K samples over 8.6K tasks, auto-generated, multi-source, more than 45 robot embodiments, and both ego and third-person human views, with diverse visual alignment.
• Cross-task protocol matters: Baselines are post-trained only on the 43 seen RoboTwin tasks, following the standard cross-task setup, while the evaluated tasks never appear as robot demonstrations during training.

Methodology in Plain English

The system works in three stages.

1. Build a task-balanced robot data pool. The authors take five public robot video-action datasets (AgiBot, InternData-A1, Open-X-Embodiment, RoboCOIN, RoboMIND) and re-partition them into tasks, where a task is defined by the combination of manipulation action and object. Instead of sampling trajectories in proportion to how often they appear, they cap how many trajectories each task contributes, so repeated teleoperation of a few tasks does not dominate. This yields more than 6,000 tasks and roughly 400K trajectories per training epoch (Task-diverse VA).

2. Manufacture human "instruction" videos automatically. For sampled robot videos, a vision-language model (Gemini 3.1 Pro or Qwen3.6-Plus) extracts task-level information — task name, initial object states, state changes, final states — and writes an image-editing prompt. An image editing model (Nano Banana 2 or Qwen-Image-2.0) turns the first robot frame into a plausible human scene, adding variation in background, viewpoint, environment style, object instance, and object placement. The VLM then writes a video-generation prompt describing how human hands should manipulate the objects, which a video model (Wan 2.7 or Kling AI 3.0) turns into a human manipulation video. Finally, the VLM filters generated videos for semantic preservation and physical plausibility, and passing videos are paired with the original robot trajectory — which keeps the executable actions — to form an ICL pair. This produces HumanGen: 74.2K pairs over 8.6K tasks, split into Pre-train ICL (External), Pre-train ICL (In-house), Simulation ICL, and Real-world ICL.

3. Train a causal video-action model that must use the video. The model is built from Wan-2.2-TI2V-5B, converted from bidirectional image-to-video generation into a causal policy. At each chunk index it predicts the next robot video chunk and then decodes the aligned action chunk, with video and action handled by separate Transformer parameter sets under a Mixture-of-Transformers design that lets them interact through shared attention. The human video is prepended as prefix memory for video prediction, and is tagged with a RoPE height-axis offset (Δ_H = 32) so the model does not confuse human latents with robot latents. Crucially, the action branch never attends to the human video directly: task information from the human video must flow through the predicted future robot video chunk, and action decoding behaves like standard inverse dynamics.

The shortcut problem is addressed by the IFP objective. If the model can predict the immediate next chunk from recent robot history alone, it can lower training loss while ignoring the human video — and then underuse the video exactly when it is needed for an unseen task. IFP therefore adds K = 4 auxiliary modules that predict strided future chunks (stride s = 2) from a fused representation of multiple intermediate layers of the main video Transformer, with loss weights (0.5, 0.25, 0.15, 0.15). These modules are not conditioned directly on the human video, so the only way to reduce the IFP loss is for the main branch to actually encode the in-context task information. The IFP modules are discarded at inference.

Training details: AdamW with peak learning rate 1×10⁻⁴ and weight decay 0.01; Task-diverse VA and HumanGen sampled at a 1:5 ratio; for non-ICL samples the language instruction is dropped with probability 0.1; for ICL samples the human-video latent is dropped with probability 0.1 and language dropout is raised from the Wan-2.2 default of 0.1 to 0.4; robot video chunk size randomly sampled from 1 to 4; 15,360 GPU hours of pre-training; maximum 160K tokens per GPU. RoboTwin post-training uses 64 GPUs for 4,000 steps with a Task-diverse VA : HumanGen : RoboTwin ratio of 2:10:3. At inference the model runs in language-only mode (video CFG scale 5) or ICL mode (human video prompt, language disabled, ICL classifier-free guidance scale 5), with inference chunk size fixed to 2 and action CFG scale 1.0.

Why This Matters

Impact on research. The paper reframes robot cross-task generalization as a task-specification problem, mirroring in-context learning in large language models. It argues against language as the sole task interface and provides a scalable recipe for generating the paired human-robot data that previous work (MIME, EgoMimic, BC-Z, RH20T, EgoScale, EgoHumanoid) had to collect manually. It also names and directly attacks a failure mode — policies that learn shortcuts from robot history and silently ignore their own in-context prompt — that is easy to miss in benchmark numbers.

Real-world applications:

• Household and service robots that a non-expert can teach a new chore by recording a short human demonstration on a phone, with no retraining.
• Warehouse and logistics manipulation, where new item types and packaging layouts appear constantly and text instructions cannot describe the required grasp and placement geometry.
• Manufacturing and assembly, especially fine-grained insertion tasks, where the paper demonstrates generalization to precision-demand insertion and long-horizon sequences.
• Flexible lab or kitchen automation, where a human video of a procedure can transfer to a robot arm handling unseen objects and containers.

Industry relevance. The human-video interface removes a major data bottleneck: human video is dramatically cheaper to obtain than robot teleoperation, and the automatic pipeline means companies do not need to stage matched human-robot collection sessions for every new task. The model also supports language and video instructions within one policy, so existing text-prompted workflows are not abandoned. The reported pre-training cost (15,360 GPU hours) and the reliance on large generative models for data synthesis do, however, imply substantial compute and third-party model dependencies.

Future Directions

• Close the reported result gap in the simulation table. The truncated content does not show the per-task numbers for place empty cup and stack blocks three or the macro-average row, so the full per-task breakdown behind the 46.95% figure remains to be seen in the complete paper.
• Extend and quantify real-world evaluation. The paper reports that Zero-WAM outperforms LingBot-VA across three real-world task families but does not report per-task success rates for them in the provided content; systematic real-world numbers, seeds, and rollout counts would be a natural next step.
• Reduce dependence on generated human videos. Because the pipeline depends on VLMs (Gemini 3.1 Pro, Qwen3.6-Plus), image editors (Nano Banana 2, Qwen-Image-2.0), and video generators (Wan 2.7, Kling AI 3.0), it is an open question how much of the performance comes from the quality of the generated videos versus the IFP objective. Ablating generation quality, or learning directly from unpaired in-the-wild human video, would test generality.
• Understand and extend the shortcut-suppression mechanism. IFP is training-only and conditioned solely through a fused multi-layer representation; how to choose the number of future chunks, the temporal stride, and the loss weights for new domains, and whether stronger objectives exist, are open questions.

Target Audience

• Robotics and embodied-AI researchers working on manipulation policies, vision-language-action models, and video-action models.
• Practitioners building general-purpose robot systems who need an instruction interface that works on tasks the policy was never trained on.
• Dataset and data-engine researchers interested in automatically synthesizing semantically matched cross-embodiment data at scale.
• Graduate students and advanced engineers familiar with Transformers, flow matching, and robot learning who want a concrete example of in-context learning applied outside language.

Authors’ abstract

Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.