The Pulse
NVIDIA details two-prompt workflow for physical AI training data
NVIDIA outlined a way to set up and run physical AI data workflows through two prompts to a coding agent. Its Cosmos Data Factory offers five reference blueprints for tasks including driving scenarios, rare-event videos and defect images.

Developers can use two prompts to set up and run workflows that prepare training data for physical AI, including synthetic driving scenes and defect images. NVIDIA's November 18 technical post describes the approach for Cosmos Data Factory, using a coding agent to configure the environment and carry out a selected task.
The system consists of five cloud-native reference blueprints and six composable skills for curating, processing and generating data. The walkthrough covers ways to expand existing datasets as well as create examples of events or defects that may be difficult to capture.
From setup to a dataset
A first prompt tells the coding agent to clone the orchestration repository and run preflight and setup. Preflight checks the environment without changing it and flags missing requirements; setup then prepares the environment to run the blueprints.
A second prompt selects a blueprint and specifies the training-data job. The orchestration skill coordinates workflow steps, deploys required services and manages execution on the user's cluster.
The post gives examples that use a developer's images or videos as inputs and specify an output location. Depending on the blueprint, the workflow can generate variations, check them and attach labels for training.
Driving and video variations
For autonomous vehicles, a blueprint searches a database of driving scenarios and generates datasets for model training and validation. Cosmos Curator extracts metadata from the input videos, while Cosmos Dataset Search finds relevant footage through embeddings.
The retrieved videos pass through Cosmos3-Super-Image2Video style transfer, and Cosmos Evaluator checks them for physical plausibility. An example request asks for scenes matching an input video to be transformed into rainy, snowy and foggy conditions while preserving layout and motion.
Another blueprint creates variations of cropped images of people, changing attributes such as clothing, accessories and color. A language model turns selected attributes into structured prompts; the workflow checks for the requested clothing attributes and labels the results with attributes and search queries for person search and re-identification.
The video augmentation blueprint changes weather, lighting or environmental conditions in existing footage while preserving scene structure. Cosmos3-Nano generates the variant, and the evaluator checks for added motion artifacts, the requested conditions and physically plausible motion. The workflow then labels both source and generated videos with detections, tracking and captions.
Rare events and defects
For rare events, a separate blueprint starts with an image and generates videos of scenarios such as falls, fires or shoplifting. A vision-language model combines a description of the scene with the requested event, and Cosmos3-Super-Image2Video generates a clip that preserves the source viewpoint and layout.
The generated clips are checked for the requested event and physically plausible motion, with failed clips retried. Clips that pass receive detections, tracks, captions and semantic annotations for downstream training.
A fifth blueprint produces defect examples from clean inspection images or 3D USD assets of parts such as circuit boards, glass panels and metal surfaces. For 3D assets, Isaac Sim renders inspection images and masks, while Qwen-Image-Edit creates clean reference images known as golden images.
Cosmos 3 Super uses those references, defect masks and requested defect types to generate defects while preserving the surrounding part. The model ships fine-tuned for printed circuit boards, glass and metal, with further fine-tuning skills for other uses. The outputs include clean and synthetic defect images, along with manifests linking results to their source and generation settings.
The blueprints are built from the Auto Labeling, AnomalyGen, Augmentation, Curation and Retrieval, Simulation, and Orchestration skills. The post says those skills can be used separately or combined into custom workflows.
Sources
- Original source · Blog post NVIDIA Technical Blog developer.nvidia.com