The Pulse
NVIDIA Adds Four Open-Weight Cosmos 3 Checkpoints for Interactive World Simulation
NVIDIA has added four open-weight checkpoints to its Cosmos 3 model family for interactive world simulation, including models for driving and two-arm robotics. All four have 16B trainable parameters and use OpenMDW1.1, which permits commerc
AI.info Team · Written with AI · fact-checked ·
NVIDIA has added four new open-weight checkpoints to its Cosmos 3 model family for interactive world simulation. In a post published on November 5, the company said the models generate multi-sensor, multi-view driving scenes or predict video from robot or camera actions. All four are post-trained examples of Cosmos 3 Nano with 16B trainable parameters.
The models use OpenMDW1.1, which permits commercial and non-commercial use under its terms. The four models are Cosmos3-Nano-Transfer-Auto, Cosmos3-Nano-Sim-Auto, Cosmos3-Nano-Sim-Transfer and Cosmos3-Nano-Sim-Bimanual.
Cosmos 3 links driving controls to sensor data
Transfer-Auto generates synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB–LiDAR generation. Per-camera World Scenario Map (WSM) controls specify scene structure and traffic motion, while text prompts guide appearance. Optional initial RGB images can anchor the scene’s appearance, and joint LiDAR generation additionally uses aligned HD-map range-map controls.
Sim-Auto is the causally distilled streaming variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps, using generated history and updated driving controls. Closed-loop interaction requires a simulation runtime and a WSM renderer to update those controls from driving inputs.
Two simulation models cover rendering and robot actions
Sim-Transfer takes spatial controls from a physics simulator and a text prompt and generates color observations that a policy can consume. Its controls include depth, segmentation, edges and blur, while text guides appearance. Physics and ground-truth state stay with the simulator.
Sim-Bimanual simulates how an environment evolves in response to a bimanual robot’s actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks, and one control sequence can switch between viewpoint changes and manipulation. Its mixed-action example uses 59 control values, padded to 64 during training.
Streaming models predict in chunks
Cosmos 3’s Mixture-of-Transformers backbone connects separate text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt, available controls and generated history. Temporal-causal attention blocks future target observations, while a key/value cache retains attention information for the next prediction.
The model operates on compressed video representations called latents; after the initial image, each latent time step represents four RGB frames. Transfer predicts one step per chunk, while the optimized Bimanual setup in the report predicts two. Smaller chunks let new controls affect generation more often.
Four training stages prepare models for adaptation
The post-training recipes cover bidirectional adaptation, teacher forcing, consistency distillation and self-forcing. In the distillation stage, a frozen teacher trains a student to generate each chunk in four denoising steps. Self-forcing trains on the student’s generated history, since a slightly mispredicted object becomes context for later frames, so errors can compound.
The recipes start from Cosmos3-Nano or compatible base weights and cover all four stages. For a new sensor rig or domain, Transfer-Auto is adapted first and then distilled into Sim-Auto, through causal training and few-step distillation. For Sim-Bimanual, the recipe includes data validation, stage configurations, checkpoint handoffs and export.
For Sim-Bimanual, running the published vLLM-Omni setup requires Linux, an NVIDIA GPU, Docker and NVIDIA Container Toolkit; it was tested on one GB200, and NVIDIA says this is not a minimum-memory specification. NVIDIA advises evaluating quality on held-out episodes, noting that a successful short run verifies execution, not convergence.