Pelican-Sim 1.0: A General World Model Simulator for Embodied Intelligence

AI summary

Overview

Research area: Robotics and embodied AI, specifically action-conditioned world models (learned simulators that predict future camera observations from robot actions).

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, flow matching, and mixture-of-experts architectures, though its core ideas can be understood without that background.

Scope: A technical report describing Pelican-Sim 1.0, a single world-model simulator trained on roughly one million robot trajectories that predicts future video from an initial image and robot joint commands, and its use in four downstream robotics applications.

What This Paper Is About

Robots need a way to predict what will happen before they act, and building that capability across many different robot types is hard because joint angles, grippers, and hand designs vary widely. This paper builds a single video-prediction model that takes a starting camera image plus a sequence of robot commands and generates a plausible video of what the robot would do next. The goal is a general-purpose learned simulator that can stand in for real-world trial and error when generating training data, evaluating policies, or improving them.

Key Contributions

1. Unified cross-embodiment action conditioning. A single 28-dimensional action vector encodes arm joints, gripper opening, and dexterous-hand joints for both left and right sides, letting one model consume data from single-arm, bimanual, gripper-equipped, and hand-equipped robots. This numerical action is paired with a camera-aligned, URDF-rendered whole-arm skeleton video, and the two are injected into alternating blocks of the same Video Diffusion Transformer. The interleaved design beats late fusion by 0.904 dB PSNR on AgiBotWorld Beta.

2. Sparse mixture-of-experts backbone for heterogeneous dynamics. Each feed-forward block in the Video DiT is replaced with a token-choice sparse MoE layer (one shared expert plus eight routed experts, two selected per token), expanding capacity for diverse robot dynamics while letting all tokens share a common pathway. Under identical training data, this reduces FVD by 6.530 relative to a dense backbone on AgiBotWorld Beta.

3. Efficient autoregressive rollouts. Causal adaptation plus few-step distillation yields a four-step autoregressive simulator, a 5.67x speedup over the 35-step model on a 21-frame benchmark, which matters because downstream decision-making requires many repeated queries.

4. Comprehensive evaluation and open-source release. Best results on all five video-quality metrics and the highest adapted EWMBench score on each of AgiBotWorld Beta, RoboMIND, and RoboTwin, plus four demonstrated downstream applications. Model checkpoints and inference code are slated for release.

Main Findings

• Video prediction quality: PSNR beats the strongest evaluated baseline by 4.636 dB on AgiBotWorld Beta, 2.080 dB on RoboMIND, and 10.343 dB on RoboTwin. On RoboTwin, FVD drops from 26.60 (Ctrl-World, strongest baseline on that metric) to 4.98. The adapted EWMBench DYN score rises by 0.426 on RoboTwin.

• Complementary action representation works better than either half alone: Combining numerical action values with rendered skeleton videos outperforms numerical-only or visual-only conditioning, and interleaving the two pathways inside the transformer beats fusing them at the end (PSNR +0.904 dB on AgiBotWorld Beta).

• MoE absorbs the action modality: The sparse MoE layers both add capacity for heterogeneous dynamics and reduce conflict between modalities, producing the 6.530 FVD improvement over the dense backbone.

• Generated data improves real policies: Adding 500 generated trajectories to 50 real demonstrations per task raises policy success on RoboTwin from 70% to 93%.

• The simulator can rank and evaluate policies: Paired with a fine-tuned vision-language model evaluator and 1,000 task-specific adaptation rollouts, policy evaluation reaches a Pearson correlation of 0.994 with actual performance across five checkpoints.

• Action selection and policy improvement both gain: Relative success improvements of 47.7% for action selection (over single-sample execution) and 20.3% for policy improvement (over supervised initialization).

• Generalization is demonstrated qualitatively across five shift types: reversed trajectories, edited scenes and object variants, held-out robot embodiments, and novel viewpoints. The paper's comparison table notes that no prior method reports all of these categories.

Methodology in Plain English

The system is built on an existing latent video diffusion model (Cosmos-Predict 2.5) with 28 transformer blocks. Given an initial RGB frame and a sequence of robot configurations, it learns to generate the subsequent frames.

The distinctive move is how robot commands are fed in, using two synchronized representations of the same underlying motion:

• Numbers. Every robot is described by a fixed 28-slot vector per frame. The left 14 slots cover seven arm joints, one gripper opening value, and six hand joints; the right 14 slots mirror this. Unused slots are zero-filled, so a simple two-finger gripper and a multi-fingered dexterous hand both fit the same interface.

• Pictures. The same commands are converted into the robot's native joint order, passed through forward kinematics using the robot's URDF description, projected through the actual camera calibration, and rasterized into a texture-free skeleton video showing joints and links, with color-coded gripper states. This makes the motion visible in image space instead of forcing the model to infer it from numbers.

These two signals enter at alternating depths: the numerical embeddings provide scale-and-shift modulation inside odd-numbered blocks, while the skeleton video runs through a parallel branch of context blocks whose outputs are added as residuals after even-numbered blocks. Training uses a flow-matching objective where only the future frames are noised and the conditioning frame is kept clean.

To handle the diversity of robot dynamics, each block's feed-forward network becomes a sparse MoE layer, routing each token to its top two of eight learned experts while always passing through one shared expert. Inference is then made fast by adapting the model causally and distilling it from 35 sampling steps down to four.

Evaluation proceeds in stages: raw video quality metrics on three held-out datasets (AgiBotWorld Beta, RoboMIND, RoboTwin), ablations isolating each design choice, then four downstream tasks on RoboTwin using a separately fine-tuned VLM evaluator that scores imagined rollouts for success.

Why This Matters

Impact on research. Most action-conditioned world models are tied to specific robots, camera setups, or action formats. This work argues that the bottleneck is the interface — how actions are presented to the model — rather than raw model scale, and shows that pairing exact numerical configurations with explicit image-space motion geometry improves both controllability and visual fidelity. The reported cross-embodiment and out-of-distribution generalization results also push the field toward shared simulators rather than per-robot ones.

Real-world applications:

• Training data amplification. Generating plausible rollouts to augment small real demonstration sets, directly addressing the cost of collecting robot data.
• Safe policy screening. Evaluating and ranking candidate policies in imagination before running them on physical hardware, catching failures without risking equipment.
• Runtime action selection. Sampling several candidate action sequences, imagining their outcomes, and executing the one the evaluator scores highest.
• Closed-loop policy improvement. Using imagined interaction as a reward signal to refine policies without additional environment interaction.

Industry relevance. Humanoid and general-purpose robotics companies face a fragmented landscape of arm designs, grippers, and end effectors. A simulator with a single shared action interface and architecture reduces the engineering cost of supporting new hardware and shortens the iteration loop between policy development and physical deployment. The planned open-source release of checkpoints and inference code lowers the barrier to reproducing and extending the results. The authors also state that Pelican-Sim 1.0 will serve as the simulation module in future versions of the Pelican-Unify model family, indicating a broader integration roadmap.

Future Directions

• Closing the kinematic consistency gap. The skeleton video is rendered from nominal forward kinematics, but nothing forces the generated RGB video to obey those kinematics. Whether predicted videos remain physically plausible for contact-rich tasks is unresolved.
• Quantitative out-of-distribution benchmarking. The comparison table explicitly notes that OOD entries summarize method-specific evaluations under non-shared protocols. A common benchmark for trajectory, viewpoint, scene, and embodiment shifts is needed to compare claims fairly.
• Scaling and long-horizon stability. The four-step distilled simulator is evaluated on 21-frame benchmarks; whether autoregressive rollouts stay coherent over the long horizons that planning and policy improvement actually require is an open question.
• Extending beyond manipulation. The unified 28-dimensional action layout covers arm joints, grippers, and hands. Adapting it to mobile bases, legged locomotion, or whole-body humanoid control would test the interface's claimed generality.

Target Audience

Robotics and embodied-AI researchers working on world models, action-conditioned video generation, or cross-embodiment learning will find the action-interface comparison and ablations most useful. Practitioners building robot learning pipelines will care about the data generation, policy evaluation, and policy improvement results, and about the single unified interface for heterogeneous hardware. Engineers evaluating diffusion transformers with mixture-of-experts layers for video will find the architecture and efficiency sections relevant. Readers without background in diffusion models or robot kinematics will need to consult the cited prior work first.

Authors’ abstract

In this technical report, we propose Pelican-Sim 1.0, a general world model simulator for embodied intelligence that predicts future observations from visual context and robot actions to support downstream learning and decision making. The model incorporates four key design features: (1) Unified action representation: a 28-dimensional action value space covering most mainstream embodiments, keeping one model valid across heterogeneous devices. (2) Action-visual injection: URDF- and camera-rendered action videos bridge actions and pixels, giving markedly better controllability across embodiments, scenes, and tasks (PSNR +0.904 over alternative fusion baselines). (3) Sparse mixture-of-experts (MoE): sparse MoE layers add capacity for heterogeneous dynamics and absorb the action modality while reducing inter-modality conflict (FVD -6.530 vs. the dense backbone). (4) Efficient rollout generation: causal adaptation and few-step distillation yield a four-step autoregressive simulator, achieving a 5.67-fold speedup over the 35-step model. Benefiting from these designs, we train on approximately one million real-world and simulated trajectories and obtain large gains in action controllability and video quality: PSNR improves over the strongest evaluated baselines by 4.636 on AgiBotWorld Beta, 2.080 on RoboMIND, and 10.343 on RoboTwin, with the adapted EWMBench DYN score up 0.426 on RoboTwin. Relying on this, four downstream applications on RoboTwin succeed: 500 generated trajectories added to 50 demonstrations per task raise policy success from 70% to 93%; policy evaluation reaches a Pearson correlation of 0.994 across five checkpoints; and relative success gains reach 47.7% for action selection and 20.3% for policy improvement. Qualitative generalization across trajectory, scene, object, embodiment, and viewpoint shifts highlights its potential as a general-purpose world model simulator.