The Pulse
NVIDIA Releases Cosmos3-H-Dreams for Surgical Robot Simulation
NVIDIA released Cosmos3-H-Dreams, a generative simulator trained on clinical cholecystectomy recordings from CMR Surgical’s Versius platform. The system generates surgical scenes in response to robot actions and runs interactively on a sing
AI.info Team · Written with AI · fact-checked ·
Twenty-eight controls drive each surgical scene
NVIDIA’s new surgical simulator takes 28 dimensions of robot input to generate a changing view of an operation. Released on October 20 as Cosmos3-H-Dreams, the system is trained on clinical cholecystectomy video and robot movements recorded with CMR Surgical’s Versius platform. NVIDIA says it runs interactively on a single RTX PRO 6000 GPU, using instrument actions to shape the video it generates. NVIDIA’s October 20 announcement
The 28 action dimensions include pose and gripper information for both arms, plus energy controls and thumbstick inputs. The training footage is 832 by 480 pixels, with robot actions aligned at 10 Hz; NVIDIA distinguishes that recording rate from the simulator’s runtime generation speed. The selected footage focuses on two instruments: a Cadiere grasper on the left and a monopolar hook on the right. NVIDIA’s model and training details
From tabletop suturing to clinical procedures
The release extends an earlier Cosmos-H-Dreams simulator, which generated interactive scenes for tabletop suturing. That model learned visual dynamics from synchronized video and robot kinematics rather than explicitly modeling tissue deformation, geometry and physical interaction. The new version shifts the task to cholecystectomy, where instrument movement can stretch tissue, obscure anatomy or alter reflections and visibility. NVIDIA’s account of the earlier model and new release
Cosmos3-H-Dreams builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment, then specializes a teacher model using curated surgical video and kinematics. NVIDIA describes the result as a real-time generative simulator trained on clinical gallbladder-removal recordings that supports live kinematic control. That “first” claim is NVIDIA’s own: the company qualifies it with “to our knowledge.” NVIDIA’s release and model description
Two checkpoints trade visual quality for speed
NVIDIA released student models using four and two denoising steps for each generated video block. The two-step checkpoint is trained separately, rather than being the four-step model run with fewer steps. NVIDIA says fewer steps speed up inference but reduce visual quality, giving developers a choice between those trade-offs. The company does not give a specific runtime frame-rate figure in its announcement. NVIDIA’s checkpoint and inference details
The model’s training process first extends the teacher’s predicted horizon from 24 future frames to 72, or 7.2 seconds at the 10-Hz action rate. Distillation then trains a causal model to produce video in short blocks as new actions arrive. NVIDIA also says it uses FlashDreams for streaming inference and an adapted FadeMem module to manage the model’s attention history during longer sessions. NVIDIA’s training and runtime description
Realistic video still needs surgical validation
The generated scenes are meant to respond to different instrument movements, including trajectories not present in the training examples. But visual plausibility alone cannot establish that the simulator gets tissue mechanics or anatomy right. The NVIDIA authors write: “Visual realism is therefore a starting point for validation, not proof of biomechanical accuracy.” Lukas Zbinden, Javier Gamazo, Nigel Nelson, Mostafa Toloui and Sean Huver, NVIDIA
NVIDIA calls Cosmos3-H-Dreams a research simulator, not a validated system for patient care. Its stated next steps include evaluating anatomy, instrument behavior and tissue response with surgeons, expanding beyond two controlled instruments and covering more of the procedure. The open question is whether those assessments can show that the generated scenes remain dependable when the simulated movements become more complex. NVIDIA’s limitations and planned research directions