The Pulse
AWS Adds NVIDIA MIG GPU Partitioning to SageMaker Inference Endpoints
Amazon SageMaker Inference now supports NVIDIA MIG partitions on endpoints that use inference components. AWS tested an H100 partition against a full A100, finding that performance and cost results varied with workload and concurrency.
AI.info Team · Written with AI · fact-checked ·
An ml.p4d.24xlarge instance with eight A100 GPUs can provide 24 GPU partitions in the example SageMaker layout. Each of the eight A100s carries three partitions: one mig-4g.20gb, one mig-2g.10gb and one mig-1g.5gb.
NVIDIA MIG splits a GPU into as many as seven isolated instances
The feature uses NVIDIA Multi-Instance GPU (MIG) to split a physical GPU into independent partitions. Each partition receives a fixed share of the GPU’s compute units and a dedicated block of its memory, with isolation enforced in hardware. Depending on the accelerator, MIG can divide a GPU into as many as seven instances.
GPU partitioning works on endpoints that use inference components, which let users deploy several models to one endpoint and set the compute each model receives. Single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned. AWS says each model can keep its own partition and stay loaded, with no load penalty or eviction.
Two optional fields configure the endpoint
AWS adds two optional fields to API calls already in use, without adding new operations, resource types or AWS Identity and Access Management (IAM) actions. AcceleratorPartitionLayouts goes on a production variant and declares how each accelerator is divided. AcceleratorPartitionType goes on an inference component and selects which partition the model runs on.
Each component copy occupies one partition of its requested type; raising CopyCount places more copies on free partitions. Partitioning happens as the endpoint is provisioned, and in AWS’s testing, carving accelerators added roughly 17 seconds to provisioning. The rest of the time is the usual instance launch and container pull.
An H100 partition was faster than a full A100 in AWS’s test
AWS compared one H100 mig-3g.40gb partition with one uncarved A100-40GB, running Llama-3.1-8B-Instruct on production SageMaker real-time endpoints. The test swept five traffic shapes, from short prompts to long-context summarization, across a concurrency ladder from 2 to 512. The H100 partition delivered 53 percent to 89 percent more output tokens per second at the load where each scenario’s cost benefit was largest.
Inter-token latency was 8.9 percent to 42.1 percent lower, and time to first token was 5.6 percent to 47.5 percent lower. The partition cost 18–34 percent less per million output tokens above the break-even crossover; that point came at concurrency 64 for 100-token prompts and 4 for 8,000-token prompts. Below the crossover, the full A100 was the cheaper unit.
At the top of the concurrency ladder on 6,000- and 8,000-token workloads, the A100 failed a third to a half of its requests against a serving-path deadline, while the partition failed none. AWS describes the comparison as workload-specific: the full A100 remained cheaper for short-input or low-concurrency workloads.
Layouts are fixed, and instance billing stays the same
The endpoint’s partition layout is chosen at creation, because MIG allows layouts to change only while partitions are idle. A component’s partition type is fixed for that component’s life, and changing its profile requires a new component. Changing the endpoint’s carve requires a new endpoint and a traffic migration.
AWS says customers are billed for the full GPU instance; partitioning increases how much of that instance they can use without changing its hourly price. AWS also says the GPU isolation does not extend to the whole instance: models on the same instance still share host CPU, system memory, storage, networking and endpoint routing.