The Pulse
AWS Shows 10-Second Recovery for GPU Training Faults
AWS and NVIDIA demonstrate a fault-tolerant PyTorch training setup on Amazon EKS using NVIDIA Resiliency Extension. The approach cuts tested recovery time from 270 seconds with standard Kubernetes restarts to about 10 seconds for in-process
AI.info Team ·
AWS says distributed GPU training jobs on Amazon Elastic Kubernetes Service can resume in seconds rather than minutes after a worker failure, but the improvement depends on replacing Kubernetes-level restarts with recovery mechanisms inside the training process itself.
In a post published September 16, AWS Specialist Solutions Architect Aravind Neelakantan and NVIDIA AI Architect Shreya Gupta describe an integration of NVIDIA Resiliency Extension, or NVRx, with PyTorch Fully Sharded Data Parallel training on Amazon EKS. Their test compared NVRx recovery with a conventional Kubernetes restart and found a sharp gap: about 10 seconds for NVRx in-process recovery, 17 seconds for NVRx's job-level launcher, and 270 seconds for the Kubernetes baseline.
The results come from controlled fault injection rather than a production incident. The authors inserted five deterministic faults into each 2,000-step training run, using two p5.48xlarge nodes with 16 H100 GPUs and checkpointing every 500 steps.
Why a single failed rank can stop the whole job
Large distributed training jobs run synchronously, so one failed worker can force healthy workers to wait. AWS describes a failure sequence in which one rank crashes, surviving ranks hit a 60-second NVIDIA Collective Communication Library timeout, and Kubernetes then restarts pods out of sync. The resulting CrashLoopBackOff and additional timeout cycles extend the interruption well beyond the original fault.
That design creates a costly mismatch between the failure and the response. A transient exception or communication hang may require only a process-level reset, while a worker killed by an out-of-memory condition or operating-system failure needs a new worker. Treating both cases as a full container restart discards more state and keeps more GPUs idle.
NVRx separates soft faults from hard crashes
NVRx supplies three independent mechanisms. Its asynchronous checkpointing layer moves checkpoint writes into a background process, allowing the training loop to continue while each FSDP rank writes its own shard. Its in-process wrapper catches transient exceptions and watchdog-visible NCCL hangs, aborts the affected process group, checks GPU, NVLink and network health, and rejoins surviving workers from the latest checkpoint.
The second recovery layer uses NVRx's ft_launcher. A heartbeat monitor watches each rank and can terminate stalled survivors, reclaim GPU memory, establish a new rendezvous and spawn fresh workers after events such as SIGKILL, an out-of-memory kill or an operating-system-level hang. The workers then reload the latest checkpoint.
AWS presents the layers as separate responses to different failure scopes: the Python wrapper handles faults that remain inside the process, ft_launcher handles failures that kill or wedge the process, and the cluster orchestrator remains responsible for node loss.
The benchmark favors recovery inside the job
In the fault-recovery test, NVRx in-process restart delivered 31% training goodput and 87% infrastructure goodput. The ft_launcher configuration delivered 25.5% training goodput and 85.9% infrastructure goodput, while the Kubernetes restart baseline reached 11.5% training goodput and 35.8% infrastructure goodput.
Goodput measures useful training progress after accounting for interruptions. The NVRx numbers do not mean the system continued training normally during failures; they show how much more of the run produced useful work after recovery overhead was included.
The test uses Llama 3.1 8B with FSDP and a fixed five-fault schedule. AWS does not present the results as a guarantee for every model, storage system or cluster configuration. Recovery time also depends on checkpoint loading, which the authors identify as the dominant delay at scale rather than the restart mechanism itself.
Async checkpoints remove a different bottleneck
The post also measures the cost of saving checkpoints. AWS tested Llama 3.1 8B across two to eight nodes, or 16 to 64 H100 GPUs, with checkpoints every 1,000 steps. Asynchronous checkpointing maintained between 99.2% and 99.8% training efficiency across that range, while synchronous checkpointing remained between 57% and 61%.
The difference comes from storage time. AWS measured roughly 275 seconds for the FSx for Lustre write path, a delay that stayed broadly constant as the cluster grew from 16 to 64 GPUs. Synchronous saves force all ranks to wait, while NVRx overlaps the write with the next training segment.
More frequent saves expose the trade-off. At eight-node scale with a checkpoint every 100 steps, synchronous checkpointing fell to 14.7% efficiency, compared with 29.6% for asynchronous checkpointing. At a 1,000-step interval, the corresponding figures were 60.3% and 99.8%.
What AWS is asking teams to deploy
The reference setup uses self-managed p5.48xlarge nodes, each with eight NVIDIA H100 80 GB GPUs and 32 Elastic Fabric Adapter network interfaces. Training pods run as Kubernetes Jobs, while Amazon FSx for Lustre provides shared checkpoint storage and Amazon ECR stores the training image.
The example requires an EKS 1.28-or-later cluster, NVIDIA and EFA device plugins, PyTorch 2.9 or later, and NVRx. The post says its benchmark uses NVRx 0.4.1, while a current deployment should use version 0.6.0 with an updated launcher configuration.
AWS publishes the Terraform modules, container setup and Kubernetes manifests through its reference implementation. The practical change is narrower than redesigning a training system: teams can add asynchronous checkpointing, in-process recovery or job-level restart independently, depending on the failures they need to contain.