GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

AI summary

Overview

Research area: Robotics — world-action models (WAMs), vision-language-action policies, robot manipulation pretraining, video generation for control.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, autoencoders, inverse dynamics models, and robot imitation learning.

Scope: The paper introduces GE-Act 2.0, a world-action model whose generative and action components are pretrained from scratch on manipulation data, and it reports how zero-shot, out-of-distribution manipulation success scales with co-training data from 300 to 30,000 hours across two robot embodiments.

What This Paper Is About

World-action models predict how a scene will unfold and use that prediction to guide robot actions, which lets them learn from both action-free video and action-labeled robot interaction. Most existing systems, however, inherit an already-trained video generator and spend their effort coupling it to an action model, so the question of how to pretrain and scale a world-action model as a whole has been left open. GE-Act 2.0 addresses this by initializing every trainable generative and action component from scratch on manipulation data, and by measuring the resulting pretrained capability directly without per-task fine-tuning.

Key Contributions

1. GE-Act 2.0 architecture. A world-action model built for pretraining from scratch on manipulation data, combining a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). The CoAE provides a compact latent space, and the single-step SVP lets visual planning and inverse dynamics be pretrained separately on complementary data before being connected through end-to-end co-training.

2. Identification of the "validity gap" and the KASO method. The validity gap is a supervision mismatch that arises when a predicted future depicts a behavior different from the recorded action used as its label. Knowledge-aligned selective optimization (KASO) co-trains the model only on action-compatible predicted futures, sampled and judged by the active IDM.

3. Systematic scaling evidence for pretrained zero-shot capability. Increasing manipulation data raises success broadly across 100 real-robot tasks and two embodiments without task-specific fine-tuning, and the results reveal a strong relationship between skill coverage and success plus useful transfer to a sparsely represented embodiment.

4. Complementary language and grounding evaluation under the same OOD protocol. Experiments demonstrate fine-grained referential grounding and instruction following even when explicit commands conflict with already-committed behaviors or conventional scene associations.

Main Findings

• Data scaling drives large zero-shot gains. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D. These figures come from evaluating pretrained checkpoints directly, with no per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting conditions, and object instances.

• A sparsely represented embodiment still benefits substantially. G2-90D comprises less than 2% of the co-training data yet improves by 17.7 points, which the authors interpret as useful cross-embodiment transfer from the broader corpus.

• Gains are broad across skills, not concentrated. The scaling gains span 19/20 and 18/20 skill groups for the two embodiments respectively.

• Skill coverage tracks success. Skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success, with Pearson r = 0.80 and Spearman ρ = 0.85.

• Strong referential grounding and instruction following. Under the same zero-shot OOD protocol, the model grounds object, color, shape, and position references in at least 90% of trials. Qualitative stress tests show it follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.

• CoAE is a favorable compactness-informativeness trade-off. In a controlled action-recovery probe with a frozen encoder and the same two-layer inverse-dynamics probe on 400 samples from a visually cluttered shelf task (100 held-out samples, 100 held-out for MAE reporting), CoAE obtains an action MAE of 0.01673. DINOv3 and V-JEPA 2.1 obtain the lowest errors at 0.01273 and 0.01487; CoAE trails them by only 13–31% while using one-sixteenth as many tokens per frame. DC-AE and SigLIP errors are 54% and 88% higher than CoAE's, at 0.02573 and 0.03138.

• CoAE also wins on instruction grounding. On a caption-matching probe built from 5,000 GenieSim-Instruction episodes, CoAE achieves 97.95% accuracy (2.05% error), 24% lower error than the next-best DC-AE (2.71%) and roughly half the 4.15% error of V-JEPA 2.1. DINOv3 scores 2.75% and SigLIP 3.28%.

• Compression rate alone does not explain the result. CoAE and DC-AE share the same 64×64 per-axis downsampling, yet CoAE performs better on both probes, which the authors attribute to manipulation-domain training combined with multi-teacher alignment. CoAE and DC-AE are also the only two of the five compared representations that retain a pixel decoder, so the compact operating point costs no decodability.

• Compactness details. At the 256×384 resolution used by the SVP, CoAE maps each input frame to a 4×6 grid of 512-channel latents, corresponding to 24 tokens per frame. CoAE is initialized from the 128-channel DC-AE by transferring compatible weights, with latent width expanded to 512 channels.

• Note on scope of the provided content. The paper content available for this summary is truncated mid-way through the world-action alignment section; detailed experiment tables, per-task breakdowns, and appendix material are not present, so numbers beyond those reported above are not available here.

Methodology in Plain English

The system is deliberately decoupled into three parts, each trained where its data is strongest, then joined.

Compressing the world efficiently (CoAE). Instead of borrowing a latent space from a general video autoencoder, the team trains a control-oriented autoencoder that compresses each frame aggressively — a 64×64 spatial downsampling factor, far beyond the 8× or 16× typical of standard video autoencoders. Training uses pixel reconstruction, perceptual (LPIPS), and adversarial losses. Crucially, the latent is also aligned to three frozen visual teachers: the SigLIP 2 vision encoder used by Qwen3.5, V-JEPA 2.1, and DINOv3, which respectively supply language-aligned semantics, spatiotemporal structure, and dense visual features. The authors then test the latent with two probes: whether a small frozen-encoder probe can recover executed actions from it, and whether a network can match an observation to its correct instruction.

Predicting the future in one shot (SVP). The visual planner has two parts: a frozen vision-language model (Qwen3.5) that reads the head-view image plus the instruction and produces scene-grounded text states, and a conditional flow generator that denoises future visual latents. The generator uses a single-step MeanFlow formulation, so an entire future — multiple views, dense near-term frames and sparse far-horizon frames — is produced in one differentiable forward pass. That single step is the key enabler: because the generated future is differentiable end-to-end, an action loss can propagate back into the generator without differentiating through a multi-step sampling chain.

Turning predictions into actions (IDM). The inverse dynamics model takes the current latents, the predicted future latents, and proprioception, and outputs both a dense control-rate action chunk and sparse actions over a longer horizon using a flow-matching objective. At deployment only the dense chunk is executed; the sparse actions are auxiliary training targets. Because the IDM's loss can be computed on recorded futures alone, it can be pretrained on instruction-free robot data such as failures and deployment rollouts — data that conventional imitation objectives struggle to use.

Handling disagreement between prediction and label (KASO). Manipulation is multimodal: the same instruction admits several valid outcome modes, but a recorded demonstration captures only one. Pairing an independently generated future with a recorded action therefore creates the validity gap. KASO samples several candidate futures and co-trains only on those the active IDM judges behaviorally compatible with the recorded action, and it keeps the original pretraining losses active during co-training so that the separate pretraining is not overwritten.

Evaluation protocol. The stated methodological choice is to evaluate pretrained checkpoints directly, without task- or embodiment-specific supervised fine-tuning, under OOD conditions with held-out scenes, backgrounds, lighting, and objects, keeping the protocol fixed across scaling checkpoints. This is intended to isolate what pretraining contributes rather than what downstream adaptation contributes.

Why This Matters

Impact on research. The paper reframes world-action models as something to pretrain and scale in their own right, rather than as adapters bolted onto an existing video generator. It argues that most prior WAMs cannot pretrain an inverse dynamics model standalone on instruction-free interaction data because their multi-step generators make the gradient path too costly, and it shows that a single-step differentiable generator removes that obstacle. The validity gap is also a concept that applies beyond this system, wherever a generated visual target is paired with a recorded action label that may correspond to a different mode. Finally, the choice to report zero-shot OOD numbers with the protocol fixed across checkpoints gives a cleaner measurement of pretraining contribution than fine-tuned comparisons.

Real-world applications.

• Warehouse and logistics manipulation, where robots must pick and place diverse items under varying lighting and backgrounds rather than a fixed, curated object set.
• Shelf and retail organization, the setting used in the paper's action-recovery probe, where a robot categorizes and organizes many items.
• Household and service robotics, where instructions like placing an object relative to another object require grounding references in the current scene rather than in language alone.
• Deployment rollout learning, where a fleet's instruction-free operational data — including failures — can be folded into training through the separately pretrained IDM.

Industry relevance. The claim that G2-90D, under 2% of co-training data, improves by 17.7 points suggests a practical route for companies with a heavily imbalanced fleet: a large corpus from one embodiment can transfer to a sparsely represented one. The correlation between skill coverage and success (r = 0.80, ρ = 0.85) also gives a concrete guidance signal for where to invest data collection next — collect demonstrations in skill groups that are thinly covered. And because the evaluation deliberately excludes per-task fine-tuning, the numbers indicate what a platform can expect before any task-specific adaptation is paid for.

Future Directions

• Extend the scaling curve past 30,000 hours. The observed gains from 300 to 30,000 hours raise the question of whether the trend continues, saturates, or shifts composition — particularly whether the sparse-embodiment transfer seen with G2-90D persists or concentrates.
• Test the KASO-preserved action diversity with post-training. The paper motivates KASO by arguing that preserved action modes give reinforcement learning broader coverage of the action space, but the surfaced content does not report RL results, so this claim is a hypothesis to be tested downstream.
• Measure what fine-tuning adds on top of pretraining. The authors deliberately report zero-shot OOD numbers to separate pretraining from adaptation; running the complementary experiment — SFT on these checkpoints — would quantify how much of the 44.1% and 31.1% is a floor versus a starting point.
• Broaden embodiment coverage. With only two embodiments evaluated and one of them under 2% of the data, extending the co-training corpus to more robot platforms is the natural test of how general the cross-embodiment transfer is.
• Tighten the validity gap further. KASO filters candidates by IDM-judged compatibility; whether better filtering, more candidates, or different compatibility criteria further improve alignment remains open.

Target Audience

Robotics and embodied-AI researchers working on manipulation policies, world models, and vision-language-action systems, especially those interested in pretraining data scaling rather than downstream fine-tuning. Also relevant to generative-modeling researchers working on few-step or one-step flow and diffusion methods, since the single-step MeanFlow-based visual planner is a conditional, multi-view extension of that line of work, and to practitioners building robot foundation models who need to decide how to allocate data collection across embodiments and skill groups. Readers without a background in flow matching, autoencoders, and imitation learning will find the middle sections of the paper dense, though the abstract-level scaling results remain readable.

Authors’ abstract

World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction. Most inherit pretrained video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.