Skip to content
AI.info

The Pulse

AWS adds stalled-job recovery and checks to SageMaker HyperPod

SageMaker HyperPod can detect stalled Slurm training jobs and optionally cancel them for requeueing. AWS also introduced an EKS training validation check and a Kubernetes monitoring agent.

AWS adds stalled-job recovery and checks to SageMaker HyperPod

Stalled AI training jobs on Amazon SageMaker HyperPod can now be detected and optionally cancelled for requeueing on Slurm clusters. AWS also introduced a training validation check and a Kubernetes monitoring agent for HyperPod clusters orchestrated by Amazon Elastic Kubernetes Service (EKS), according to a company post.

The existing health monitoring agent watches individual nodes for GPU, network and kernel errors, while deep health checks test hardware and connectivity. AWS says those checks do not establish whether GPUs compute correctly during training, whether a job is progressing or whether Kubernetes components are healthy.

Testing training across instances

The new MultiNodeTrainingValidation check runs a fixed training workload first on each instance and then across instances of the same type. It is available for EKS clusters using supported NVIDIA GPU instances with Elastic Fabric Adapter (EFA) networking. The test runs when configured instances are created or updated, or on demand.

The check runs a short Megatron-LM workload on all GPUs in each instance, followed by a test across instances using EFA. Instances fail if the training loss becomes NaN or infinite, or if training does not finish within the time limit. AWS documentation also lists loss rising above a threshold relative to its starting average as a failure condition.

Failed instances are labeled and, if automatic node recovery is turned on, rebooted or replaced. The check adds time to the deep health check run, according to the documentation.

Monitoring the Kubernetes layer

A separate Kubernetes Resource Monitoring Agent watches EKS clusters for problems including device plugins that repeatedly crash, missing DaemonSets and tainted nodes. When a job fails, it reports the pods and nodes involved through Kubernetes events and Amazon CloudWatch logs.

It also monitors node condition changes and reports NCCL communication state for jobs that opt in. NCCL is the NVIDIA communication library used by the GPU connectivity tests described in the AWS post.

Detecting stalled jobs

For Slurm clusters, the new hung-job detector runs an agent on each node and watches job output and GPU activity, or user-defined log patterns. Detection is off by default and must be enabled for each job through a rules file and an environment variable. Existing clusters need their software updated to the latest AMI before the feature is available.

With default detection, the agent looks for output that has stopped while GPUs are idle but still hold memory, or for a longer pause in output regardless of GPU state. Custom log-pattern rules replace that default detection. Default detection cannot catch a job that keeps its GPUs busy without making progress, such as one stuck in a loop, AWS documentation says.

The default action records an event; a cancellation action can also be configured, with Slurm requeueing cancelled jobs by default. Hung-job detection already exists for EKS jobs run through the HyperPod training operator, while Ray Train jobs get it by default.

A cost illustration

In an illustrative AWS scenario, a pre-training job uses 32 ml.p5.48xlarge instances with 256 NVIDIA H100 GPUs and hangs from 1:00 a.m. until 9:00 a.m. AWS estimates the hang would cost about $16,900 at the HyperPod on-demand price of approximately $66 per instance-hour in US East (N. Virginia).

With a 15-minute detection rule that cancels the stalled job for Slurm to requeue, the estimated cost falls to about $530, a saving of approximately $16,400. The figures describe the company’s example rather than a measured customer result.

Sources

  1. Amazon Web Services
  2. Amazon Web Services
  3. Amazon Web Services

Companies in this story

Related stories

Explore

More articles