The Pulse
NVIDIA Adds Automated Recovery for Slurm Training Jobs
NVIDIA says its recovery service recorded about 8,550 anomaly detections on one training cluster between July 9 and October 9. The open-source tool automates recovery for Slurm training jobs, with new deployments set to monitor mode before
AI.info Team · Written with AI · fact-checked ·
One cluster logged 8,550 anomalies
NVIDIA says its recovery service recorded approximately 8,550 anomaly detections on one training cluster between July 9 and October 9. The same monitoring period covered nearly 200 jobs per day, with mean detection latency of 69 seconds and mean recovery-action latency of 94 seconds. NVIDIA cautions that the latter measurement ends when the automated action occurs; scheduling and checkpoint restoration determine when training resumes.
The figures come from NVIDIA’s production observations of NVIDIA Recovery, an open-source service for coordinating recovery across a cluster’s training workloads. The company says the service detects interruptions through application logs, GPU metrics and scheduler signals, then uses failure attribution to guide isolation and restart decisions. It is part of NVIDIA DSX OS, the company’s software portfolio for AI factory operations.
Four stages from failure to restart
NVIDIA describes the process as a four-stage loop: detect, attribute, isolate, then restart or stop. The detector looks for crashes, hangs and stragglers, including PyTorch and NCCL failures, CUDA errors and interconnect failure patterns. Missing progress signals or recognized communication-hang patterns can indicate a hang, while sustained throughput degradation against a recent baseline can indicate a straggler.
During attribution, NVIDIA Recovery classifies the interruption using scheduler signals and application logs. An optional integration with NVIDIA Attribution can add diagnoses that distinguish hardware faults, system-software issues and application bugs; operators can also connect their own attribution service. The resulting evidence informs node exclusion and can be added to the persistent audit trail.
The remediator can exclude implicated nodes from later restart attempts or stop a workload when more retries are unlikely to help. A watchdog lets normal scheduling proceed if a recovery decision takes too long or fails. When the scheduler restarts a job, the application uses its existing restart logic to resume from its last saved checkpoint; operators can configure a limit on attempts that fail to save a new checkpoint.
Slurm support starts in monitor mode
NVIDIA Recovery runs as a control-plane service and communicates with a scheduler integration through an HTTP or gRPC API. The current release supports Slurm; Kubernetes support is on the roadmap. For Slurm, NVIDIA provides a reference Advisor Controller that can be deployed with a Helm chart either inside the cluster or in SSH mode, which connects remotely without adding an in-cluster service.
New deployments start in monitor mode. The service records recommendations but takes no scheduler actions, allowing operators to compare its advice with their own recovery decisions before enabling action mode for a cluster. IBM Research used that approach on its Slurm cluster, validating the recommendations before turning on autonomous action.
An audit trail tracks each decision
NVIDIA Recovery records each recommendation before the scheduler controller acts. Its persistent audit trail includes the matched detection pattern, action, timestamp, workload identifier and any NVIDIA Attribution verdict. Recorded actions can include killing or requeuing a job, releasing it, or excluding a node.
The Recovery Console shows live workload state across scheduler types, flags anomalies and provides a timeline of detection events and recovery actions for each workload. NVIDIA says policy-engine and watchdog actions appear in the same trail. If the controller restarts during recovery, it checks the scheduler’s current state before continuing, to avoid repeating actions already completed.
Deployment and limits of the release
NVIDIA publishes the code in the dsx-ai-factory/NVIDIA-Recovery GitHub repository under the Apache 2.0 license. A single Helm umbrella chart installs the service, and its bundled MySQL sub-chart supports a local quickstart without a physical GPU; NVIDIA recommends a managed MySQL service for production. Operators connect the chart to an existing Loki or Victoria Logs endpoint and Prometheus scrape target, then run a configurator that checks telemetry against the service’s expected format.
To opt a workload in, an operator adds a directive to its sbatch submission script. NVIDIA’s instructions then call for reviewing recommendations in the Recovery Console and changing the service to action mode after validating them. The current release handles Slurm jobs, including Slurm on Kubernetes through the slurm-operator; job arrays and native Kubernetes recovery for JobSet and Kubeflow PyTorchJob are listed as future work.
Verda, a neocloud provider, deployed the service with Victoria Logs and contributed a connector to the repository, according to NVIDIA. IBM Research is also working with NVIDIA to refine failure patterns from its training workloads. The service’s immediate scope is therefore clear: it coordinates recovery for opted-in Slurm jobs, while the cluster’s scheduler and checkpoint logic still determine when training becomes productive again.