The Pulse
AWS describes checks for silent GPU errors in AI training
AWS has published a method for detecting silent data corruption in training jobs on SageMaker HyperPod. Its tests used software-injected faults and identified limitations in what the checks can catch.
Three checks added to an AI training loop can flag some silent GPU errors and help identify a suspect node for replacement, according to an Amazon Web Services technical post. AWS described the method for SageMaker HyperPod on November 11, reporting tests with software-injected faults.
Silent data corruption occurs when hardware returns a plausible but incorrect number without raising an error. In distributed training, that value can enter the gradients and weights of model replicas while the training loss can still look normal.
HyperPod already has health monitoring, deep health checks and automatic node recovery for hardware that reports faults.
Checks inside the training job
The new method puts three checks in a PyTorch DistributedDataParallel training loop, where replicas train on different portions of a batch and share gradients. It is a method for training teams to implement, not a claim that checks have been enabled automatically for every HyperPod job.
A gradient check looks for non-finite values and overflow, while a ring replay audit has each GPU recompute a small batch from another node and compare its loss and, with deterministic kernels, its gradient fingerprints. A weight fingerprint comparison checks replicas after an update for differences in their stored weights.
The distinction matters because an incorrect gradient can be shared across replicas: they can all apply the same wrong update and still have matching weights. The replay audit is intended to detect finite wrong values that change the loss, or a gradient when deterministic kernels are used, on an audit step.
Finding and replacing a node
The proposed recovery process maps a suspect training rank to its node, stops the job and prevents new work from being scheduled on that node while it is investigated. An ambiguous alarm can prompt further checks before a node is replaced and training resumes from a checkpoint saved before the suspect window.
AWS also describes a controller that a customer could run in its own cluster and account to automate parts of that workflow. For replacement, that controller would seek human approval before calling the replacement API, or use a node-health label if automatic node recovery is enabled.
Tests and limitations
The tests used software-injected faults on a HyperPod cluster with four ml.p5.48xlarge instances, totaling 32 NVIDIA H100 GPUs, and repeated the scenarios with three instances, the minimum the checks require. The model was a 953-million-parameter Llama-style decoder trained on synthetic data.
In one scenario, a persistent injected error began at step 23; AWS reported that steps 23 through 29 were applied before an audit at step 30 identified the suspect rank and stopped that step. Fault-free runs produced no alarms across 15,320 audit steps and 3,080 weight checks, according to the post.
The checks added time in a measurement using the same model and 32 GPUs. For that setup, deterministic audits every 10 steps increased average step time by 34.7% over a plain training loop, with weight checks also running every 10 steps. Default-kernel audits at the same interval increased it by 12.5%, the authors reported.
The authors reported that a backward-only weight-gradient fault produced no alarm in 50 steps with default-kernel audits. A finite fault that occurs once on a non-audit step and does not recur is not caught by the replay audit. Deterministic audits can also miss a fault confined to default kernels used on other steps.
The post lists gradient accumulation, BatchNorm, unused parameters and forward hooks with side effects among cases not tested in the example. It says the training loop and checks were tested only with DistributedDataParallel.