The Pulse
AWS Adds SOCI Lazy Loading to SageMaker Training Jobs
AWS is adding Seekable OCI indexing to Amazon SageMaker AI training and processing jobs, allowing containers to start before their full images download. In AWS benchmarks, a 13.2 GB image started in about six seconds instead of 3 minutes 42
AI.info Team · Written with AI · fact-checked ·
Amazon Web Services announced on October 27 that SageMaker AI training and processing jobs can use Seekable OCI (SOCI) indexes to start containers before downloading their full images. Instead of waiting for every layer to arrive, SageMaker fetches the files needed to start a container and streams the rest as the job uses them. AWS says the change requires no edits to a job definition, entry point or data channels.
Six seconds to start a 13.2 GB image
The delay SOCI targets comes before model code runs: SageMaker jobs ordinarily download, decompress and write the full container image to the instance. AWS says deep-learning images often exceed 10 GB, making that wait a recurring part of billable job time. It can take a larger share of short experiments, processing tasks and hyperparameter sweeps made up of many jobs.
In AWS’s benchmark, a 13.2 GB compressed image took a median of 3 minutes 42 seconds to download conventionally and about six seconds with lazy loading. Across eight images ranging from 1.8 GB to 35.4 GB, the company measured SOCI startup times of 3 to 11 seconds, compared with conventional downloads ranging from 48 seconds to 7 minutes 17 seconds. The tests used SageMaker AI jobs on CPU and GPU instances in the US West (Oregon) and US East (N. Virginia) regions.
The first epoch still pays a cost
Starting the container sooner does not mean every file is already local. AWS found that the first training epoch took 3 to 16 seconds longer with a lazy-loaded image as frameworks fetched files they encountered for the first time; later epochs ran at the same speed as with a full download, within run-to-run variation. Across the training jobs tested, AWS says the net result was a finish 1 to 6 minutes sooner.
That saving depends on what a job reads before useful work begins. If startup code scans, checksums or preloads most of the image, the first epoch may have to wait for much of the same data a full pull would fetch. AWS’s tests also found a smaller net benefit when workloads loaded large model weights or compiled code at the outset: those files still had to stream before use.
Pre-indexed images or custom builds
Users can switch to an AWS Deep Learning Container with a “-soci” tag or generate an index for a custom image and store it in Amazon Elastic Container Registry. SageMaker checks for an index when it creates a job; if no index exists, or lazy loading fails, the job uses a standard download instead. AWS says the original image digest and signatures remain valid because the index is stored as a separate OCI-compliant artifact.
AWS lists pre-indexed options for PyTorch Training, TensorFlow Training, vLLM and SGLang, with ARM64 variants for some frameworks. Custom indexing requires a Linux host with containerd and the SOCI snapshotter, plus Finch or nerdctl. AWS says support is available in all regions, but images must be in private ECR repositories; images pulled through a private Docker registry using SageMaker’s VPC repository access mode are not eligible.
Where lazy loading may matter most
The benchmark suggests the strongest case is for large images and jobs that begin work after using only a fraction of their contents. AWS reports comparatively small gains for images below roughly 4 GB. The result is a targeted change to job startup—not a faster training loop—and its practical value will depend on image size, startup behavior and how much of the image a workload needs immediately.