Capacity, Cost, Performance, and Sustainable ML Operations

Aggressive batching brings inference cost down. Average GPU utilization improves, but tail latency exceeds the decision deadline for rare long requests, causing retries and duplicate work.

Take a server that typically responds in 10ms but has a 99th-percentile latency of one second. Handled by that one server, a request is slow once in 100 — a rounding error on any dashboard. Wait on a hundred such servers in parallel and the picture inverts. Jeffrey Dean and Luiz André Barroso wrote it out in 2013: “If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second”. Make the servers a hundred times better — one slow response in 10,000 — and fan out to 2,000 of them, and almost one user request in five still takes more than a second. The tail is multiplied by fan-out. It is not averaged away by it.

The effect has been reproduced outside the paper. Three researchers at Cornell rebuilt it in a simulator validated against a real NGINX deployment, and concluded that “for cluster sizes greater than 100 servers, 1% of slow servers is sufficient to drive tail latency high, consistent with the results in [16]”. Reference 16 is Dean and Barroso.

Cost optimization should measure acceptable decisions delivered, not isolated kernel efficiency. Reliability, queue behavior, human workload, and rework belong in the denominator.

On 7 December 2021 the scaling action was the incident. AWS's own summary of the US-EAST-1 service event states the trigger plainly: “At 7:30 AM PST, an automated activity to scale capacity of one of the AWS services hosted in the main AWS network triggered an unexpected behavior from a large number of clients inside the internal network.”

Compute is visible, but several other costs determine the real system economics — and even the visible layer is not where most of the compute sits.

Serving, not training, is where the power goes. Meta published its own split in 2022: “At Meta, we observe a rough power capacity breakdown of 10:20:70 for AI infrastructures devoted to the three key phases — Experimentation, Training, and Inference”. Seven watts in ten go to serving. AWS's Well-Architected Framework puts the same weight on the invoice: “Up to 90% of the infrastructure spend for developing and running ML applications is often on inference.” Two operators with no reason to agree, one conclusion. The training run is the story. The serving tier is the bill.

The same Meta paper reports an approximate 30%/70% embodied-versus-operational carbon split for its large-scale ML tasks. Roughly a third of the footprint is already fixed in manufactured hardware before a single request arrives. A cost stack drawn only around model compute misses the layer that carries seventy percent of the power and, on the AWS figure, up to ninety percent of the spend.

Average request rate hides peaks, burst length, request size, feature latency, model heterogeneity, and warmup time. Training plans also need queueing, checkpointing, failure waste, and experiment concurrency.

Use workload units such as examples, valid tokens, images, entities, or completed batches, because a raw request count may combine tiny and enormous jobs that consume very different resources.

The unit is not cosmetic, and the gap it hides is measurable. One table in the Dean and Barroso paper reports a real Google service twice. Measured at the root, the 99th-percentile latency for one random leaf to finish is 10ms. The 99th percentile for all leaf requests to finish is 140ms. Both rows describe the same user-visible operation. A capacity plan built on the first number is not the plan built on the second, and only the second is what the user waits for.

A single “cost per request” number can reward the wrong design. Utilization, time-to-quality and cost per acceptable decision can all move in the right direction while the user waits longer. The 10ms row and the 140ms row of that Google table are one service, read with two different units. The three columns below are meant to be read together rather than ranked.

Use measured workload distributions and explicit service objectives. The two incidents in this lesson each show which step was missing. On 7 December 2021 retries fed the congestion they were a response to. On 12 June 2025 Service Control had no randomized exponential backoff to damp the herd effect its own restarts created, and Google Cloud's report says so outright. Neither gap is visible in an average-utilization chart.

A smaller model may reduce compute but increase manual review. A faster cache may serve stale features. A lower redundancy level may save money while consuming the error budget during ordinary failures.

Every anchored case above has the same shape. The number that was optimized was not the number that governed the outcome. An automated scaling activity improved capacity and produced an event whose network devices did not fully recover for nearly seven hours. A carbon estimate built on assumed hardware came out 88X too high for the operator that ran the job. A per-token price identical for every customer charges a Telugu user 5× what it charges an English one. Ninety percent of the spend, on the AWS figure, sits in a serving tier that no training benchmark measures.

Present cost decisions with quality, reliability, human burden, slice impact, and recovery capacity on the same page.