The Pulse
NVIDIA releases open-source system for restoring GPU workloads
NVIDIA introduced Snapshot, an open-source Kubernetes system that saves initialized GPU workloads for reuse. In NVIDIA’s tests on B200 GPUs, restoring replicas took 3.6 to 42.7 seconds, compared with cold starts of up to 19 minutes.
AI.info Team · Written with AI · fact-checked ·
A reusable starting point
NVIDIA introduced Snapshot, an open-source system that saves initialized GPU workloads and restores them for reuse across inference replicas. The company describes it as a component of its DSX AI Factory Platform, intended to reduce the time needed to bring inference capacity online after demand rises or a replica fails.
Snapshot is designed for GPU workloads running on Kubernetes. It records the CPU and GPU memory of a fully initialized workload in a portable checkpoint artifact; a new Pod can then resume from that saved state on a node with the same GPU model and driver. The restore skips repeated model-weight loading, compilation and graph capture, NVIDIA says.
How the process works
Platform teams can use Kubernetes custom resources to create a checkpoint of a running Pod or request a SnapshotJob that checkpoints a workload once it is ready. To restore, they annotate a new Pod with the name of the checkpoint to use. The workload needs a small handshake to pause before saving and resume after restoration, the company says.
The system was initially developed for NVIDIA Dynamo inference workers, but other inference engines, schedulers, controllers and platforms can integrate it, the company says. Compatible workloads can span multiple GPUs on a single node and be saved as one artifact across ranks, including workloads using tensor parallelism, data and expert parallelism, and decode context parallelism. Ready-made examples are available for vLLM, SGLang and TensorRT LLM.
Benchmark results
The system was tested with those three inference engines and models ranging from 600 million parameters to mixture-of-experts models with checkpoints of up to 1 TB across eight GPUs. In the company’s tests, restore times ranged from 3.6 to 42.7 seconds, which the company says was 8x to 36x faster than starting from scratch. The largest tested MoE workloads restored in under 43 seconds; their cold starts took 6 to 19 minutes.
The benchmark setup used NVIDIA B200 GPUs and VAST NFS storage, with vLLM 0.31.0, SGLang 0.5.21 and TensorRT LLM 1.3.0. Checkpoint creation took 15 seconds to 5.5 minutes depending on the model and configuration, and checkpoint creation plus one restore was faster than a single cold start in every tested configuration, the company says.
What the measurements include
The cold-start clock ran from process launch until engine initialization finished, while the restore clock ran from the restore request until the workload was serving again. Both measurements excluded pulling the container image and setting up the container runtime, so those steps remain outside the reported restore times.
The breakdown for the eight-GPU MoE tests found that restoring GPU state took 18 to 29 seconds. Restoring CPU process state with CRIU, the Linux process checkpointing tool used in the test, took 4.2 to 10.8 seconds. The repository includes benchmarking tools and reproduction instructions, along with installation and usage guides.