Online Inference Services

On 18 November 2025 Cloudflare ran this exact failure at planetary scale, and then published the postmortem. A permissions change in ClickHouse made the query that generates the Bot Management feature file emit extra rows. The Register, reporting the incident, described a change that “returned extra info that more than doubled the size of the feature file”. The enlarged file blew past a hard ceiling: “Currently that limit is set to 200, well above our current use of ~60 features”. The ceiling exists “because for performance reasons we preallocate memory for the features”. Two proxy generations were serving traffic behind that file. They failed in opposite directions. Pingdom's monitoring recorded that “The outage started at 11:20 UTC and cloud services were restored by 17:06 UTC the same day”.

A model executes in twelve milliseconds. The user waits eight hundred. Feature lookup, authentication, serialization, queueing, postprocessing, and downstream policy consume the rest of the budget.

Online serving is request-path engineering. The model is one dependency whose latency, memory, concurrency, and failure behavior must fit an end-to-end decision contract.

The rest of that budget has a measured price. Two companies injected delay into their own live traffic and watched what happened. Google reported the search experiments in 2009: “Experiments demonstrate that increasing web search latency 100 to 400 ms reduces the daily number of searches per user by 0.2% to 0.6%.” Table 1 holds the individual results. A 400 ms post-header delay cost 0.59% of daily searches per user over six weeks. A 200 ms post-header delay cost 0.29% over six weeks. A 100 ms pre-header delay cost 0.20% over four weeks, and a 50 ms pre-header delay had no measurable impact.

Two details matter more than the headline. The loss deepened with exposure: for the 400 ms delay, −0.44% in the first three weeks against −0.74% in the second three. And it outlived the fix, still −0.21% averaged over the five weeks after the delay was lifted.

Microsoft priced the same effect in money. Kohavi and colleagues published the Bing result in 2013: “We recently ran a slowdown experiment where we slowed 10% of users by 100msec (milliseconds) and another 10% by 250msec for two weeks. The results showed that performance absolutely matters a lot today: every 100msec improves revenue by 0.6%.”

At that exchange rate, the milliseconds the model did not spend are not overhead around the product. They are the product.

Using one health endpoint for all three creates bad traffic decisions. Kubernetes' own documentation refuses to collapse even the first two: “The readiness and liveness probes do not depend on each other to succeed”. The third state has no probe at all, and that is where the damage hides. Cloudflare's FL proxy on 18 November 2025 was live and ready by any process-level test. Every request it scored came back with a bot score of zero. Nothing in liveness or readiness is designed to notice that. Prediction quality has to be measured against data and model evidence, not inferred from a container that has not crashed.

A target such as 250 milliseconds must be assigned to real stages. And the target at the end of that assignment should be a percentile, not an average. The industry's own serving benchmark is written that way. In MLCommons' MLPerf Inference rules, the Server scenario is defined so that “LoadGen sends new queries to the SUT according to a Poisson distribution”. The run lasts 600 seconds. The reported metric is the maximum Poisson throughput that still meets a 99th-percentile latency bound.

The bounds are per benchmark rather than global: Llama2-70b at TTFT/TPOT 2000 ms/200 ms, tightened to 450 ms/40 ms in the interactive category; DeepSeek-R1 at 2000 ms/80 ms; DLRMv3 at 80 ms. NVIDIA restates the constraint from the vendor side: “In the server scenario, the time-to-first-token (TTFT) threshold is 2 seconds with a 12.5 tokens/second/user (TPS/user) target. All TPS/user targets are 99th percentile”. Split your own 250 ms against a percentile deadline, or you will be budgeting against a number no user ever experiences.

A retry may help a transient network fault. It also adds work during congestion, and it can turn a slow dependency into a retry storm. Bronson and colleagues put numbers on the tipping point in 2021. A web application sits steady at 280 QPS against a database that answers in under 100 ms only while it stays below 300 QPS. A 10-second network outage is enough to start it. The retries provoked hold offered load at 560 QPS, goodput collapses to zero, and the system stays there once the trigger is gone. “So long as latency is high, client queries will continue at 560 QPS due to retries.” Getting out again requires load driven under 150 QPS, or retries capped below 20 QPS. Timeouts, retry budgets, circuit breaking, and admission control are therefore one design, not four separate ones.

The service should distinguish not ready, temporarily overloaded, invalid request, dependency unavailable, and model abstention. Collapsing these states into one generic error prevents safe fallback and useful monitoring. Several of them already have standardised wire representations waiting to be used. RFC 9110, published in June 2022, defines the overloaded one: “The 503 (Service Unavailable) status code indicates that the server is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay.” The same document gives the server Retry-After to say how long — MDN's reference documentation notes that it “should contain the estimated time for the recovery of the service”. A rate-limited client gets 429 Too Many Requests instead, a status code defined not in RFC 9110 but in RFC 6585. Returning one undifferentiated server error throws away distinctions a standards body already made for you.

The envelope should preserve attribution and safe degradation under load. Two of these steps have published defaults, which beats inventing your own: argue with a number someone has already operated at scale.

Step 3, controlling load, has one from Google's SRE book: “Consider having a server-wide retry budget. For example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don’t retry; just fail the request.” The same chapter explains why the budget belongs in one place rather than at every layer. Three tiers each retrying three times “may create 64 attempts” against an already-overloaded database. Huang and colleagues state the bound from the other end: “An example of such a policy change is decreasing the maximum number of retries per request. For instance, a policy with at most two retries will not amplify the work more than three times, while the policy with no cap effectively leaves the system with no stable region.”

Step 5, recording attribution, is where Circular 2022-03 lands. The record must identify the model, the features, the policy and the timing. It must also identify which fallback ran. Otherwise the reasons you later disclose will describe a scoring path that never executed.

Model execution time can improve while end-to-end latency and decision validity deteriorate. Service objectives should measure the complete path and classify failures by cause.

Load tests should include cold start, dependency slowdown, skewed request sizes, cache miss, and retry amplification — not only steady average traffic. That list is not a matter of taste. It is what the published postmortems keep naming. Huang and colleagues went through public incident reports and counted: “First, metastable failures are universally observed—we present an in-depth study of 22 metastable failures from 11 different organizations.” Among them, “at least 4 out of 15 major outages in the last decade at Amazon Web Services”. The outages ran “in a range of 1.5 to 73.53 hours”.

And the thing that kept the systems down was, overwhelmingly, the one behaviour a steady average-load test never provokes: “By far, the most common sustaining effect is due to the retry policy, affecting more than 50% of the studied incidents”. Bronson and colleagues had reached the same conclusion a year earlier: “One of the most common failure-sustaining mechanisms is request retries.” A load test that never drives the system past its stable region has tested the only regime that was never in doubt.