Online Serving Data, Caching, and Fallbacks

The test should exercise the latency distribution and the way meaning degrades, together. Define the end-to-end budget and allocate time to network, lookup, model, policy and response. Reproduce realistic keys, including hot, cold, new, missing, high-cardinality and skewed entities. Inject dependency failures — timeout, stale replica, partial region, cache miss, delayed materialization. Measure tails, reporting p95, p99, timeout, fallback and stale-value rates rather than averages. Compare predictions, quantifying score and decision changes under every fallback branch. Then set a release gate that approves only when the latency budget and the decision-risk budget both pass.

The injection step is the one that pays. On 12 June 2025 a globally replicated quota policy containing blank fields hit an unprotected null pointer and put Google's Service Control into a crash loop worldwide. The outage ran from 10:49 to 13:49 PDT. The kill switch was fully rolled out within 40 minutes — and the outage was not over, because the recovery was now the load: “as Service Control tasks restarted, it created a herd effect on the underlying infrastructure it depends on (i.e. that Spanner table), overloading the infrastructure”. One region took far longer: “It took up to ~2h 40 mins to fully resolve in us-central-1”. Google's own incident report names the missing control without euphemism: “Service Control did not have the appropriate randomized exponential backoff implemented to avoid this.”

The same report names the test that never ran, one stage earlier: “If this had been flag protected, the issue would have been caught in staging.” A ramp against a healthy dependency exercises neither branch. Forty minutes of fault plus roughly two and a half hours of restart herd is the shape of a real failure. It only appears if step 3 fails the dependency and step 4 keeps measuring through the recovery.

Log the feature version, materialization time, retrieval time, age, fallback state, and source health for each prediction or a privacy-safe sample. These fields make training–serving skew measurable. When labels mature, join outcomes to the exact evidence trace used at decision time. Do not reconstruct online inputs from corrected warehouse tables unless the difference is intentional and documented. Trace retention should match debugging and governance needs while respecting minimization and access controls.

For a high-risk AI system placed on the EU market, both halves of this lesson stopped being good practice and became duties with article numbers. Article 12(1) of the EU AI Act, Regulation (EU) 2024/1689, is flat about the trace: “High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system.” Article 15(4), on resilience to errors, faults and inconsistencies, names the mechanism this lesson has been describing: “The robustness of high-risk AI systems may be achieved through technical redundancy solutions, which may include backup or fail-safe plans.”

For those systems the fallback branch, and the trace that records it, answer to a citation rather than to a code review.

A feature can be exactly right in the warehouse and still miss the decision deadline online. The branch that usually rescues the request — the retry — is also the branch that most often turns a ten-second problem into a multi-hour one.

Take a web app running normally at 280 QPS that meets a 10-second network outage. When the network comes back, the database traffic does not return to 280. It stays stuck at 560 QPS, pure retry amplification, and the system will not leave that state unless load drops below 150 QPS or the retry policy changes. Bronson and three colleagues worked that example in 2021: “So long as latency is high, client queries will continue at 560 QPS due to retries.” The trigger lasted ten seconds. The state it left behind sustains itself.

A later study by Huang and colleagues went looking for the pattern in public incident reports: “an in-depth study of 22 metastable failures from 11 different organizations”, with outages running from 1.5 to 73.53 hours. Its headline finding is about exactly this branch — “the most common sustaining effect is due to the retry policy, affecting more than 50% of the studied incidents”. They add that “at least 4 out of 15 major outages in the last decade at Amazon Web Services were caused by metastable failures”. Everything below is the same feature path meeting that deadline.

Offline correctness answers what a feature should mean. Serving correctness adds whether that value can be retrieved within the decision deadline and under partial failure. For each feature, specify maximum age, availability target, timeout, default or fallback, provenance, and how the model learns that a fallback was used. A system that silently substitutes stale or missing values can preserve uptime while changing the model’s input distribution.

The HTTP standard has shipped precisely this contract since May 2010. It is worth reading as an existence proof that the terms can be numbers on the wire rather than prose in a design document. RFC 5861 defines two Cache-Control extensions, stale-while-revalidate and stale-if-error, each taking an explicit delta-seconds bound. Under the first, caches “MAY serve the response in which it appears after it becomes stale, up to the indicated number of seconds”, while the refresh happens in the background. The second is granted in a single line: “The stale-if-error Cache-Control extension indicates that when an error is encountered, a cached stale response MAY be used to satisfy the request, regardless of other freshness information.”

Two properties of that design carry directly into a feature contract. First, the staleness window and the error-fallback window are separate numbers, published with the value by whoever owns the value. How long may this be old in normal operation, and how long may it be old when the origin is failing? Second, the error case is enumerated rather than left to judgement. The Mozilla Developer Network's reference for Cache-Control documents the same two directives, and records that stale-if-error applies to “any response with a status code of 500, 502, 503, or 504”. A feature contract that says only "highly available" has published neither number.

The serving path should be chosen from the decision budget and feature semantics. Request-time compute calculates the feature from live dependencies for each request: freshest possible, but slow and fragile, liable to fan out to many systems, best for cheap local logic. A precomputed online value materializes the feature before requests and stores it for low-latency retrieval: fast and stable, needs a refresh policy, can become stale, best for expensive aggregates. A cache reuses recent results under a key and an expiration policy: reduces load, requires invalidation semantics, tail behavior matters, best for repeated access. An approximation or fallback uses a simpler value when the primary feature is unavailable: improves resilience, changes prediction behavior, needs separate telemetry, best when the consequence is understood.

The cache column carries a cost that list omits. Roblox's post-mortem of its 28–31 October 2021 outage is the clearest published statement of it. The storage underneath was never the problem. Daniel Sturman wrote it plainly: “These databases were unaffected by the outage, but the caching system, which regularly handles 1B requests-per-second across its multiple layers during regular system operation, was unhealthy.” The caching system had to be redeployed and re-warmed. That redeployment stalled because the internal scheduling data the cache system keeps in Consul KV had become incorrect after the cluster snapshot reset.

A cache is therefore not a transparent accelerator sitting in front of the real system. It is stateful infrastructure with its own control-plane metadata, its own failure domain, and a recovery time set by re-warming rather than by the database. For a platform used by fifty million players a day, the bill was stated exactly: “At 16:45 Sunday, 73 hours after the start of the outage, 100% of players were given access”.

Tail latency and failure accumulate across the whole path. Request context validates the entity ID, event time, consent and decision deadline. Lookup or compute retrieves online state or calls dependencies under a timeout budget. The freshness check evaluates source age, feature age and version compatibility. The fallback decision chooses cached, default, approximate or abstain behavior according to policy. The prediction trace records values, ages, versions, fallbacks and latency for later evaluation.

The freshness check is the box that looks safest and is not. Amazon Web Services' own post-event summary for the US-EAST-1 disruption describes “a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service's regional endpoint”. The staleness check existed and ran. It simply ran on stale information: “At the same time that this clean-up process was invoked, the first Enactor (which had been unusually delayed) applied its much older plan to the regional DDB endpoint, overwriting the newer plan.” The clean-up then removed the older plan as stale, leaving nothing behind the name dynamodb.us-east-1.amazonaws.com. The event ran from 11:48 PM PDT on 19 October 2025 to 2:20 PM PDT on 20 October.

Two details belong in any design of this path. First, the fix was not the recovery. The records were restored at 2:25 AM, but clients only saw correct evidence “as cached DNS records expired between 2:25 AM and 2:40 AM”. The cache expiry, not the repair, decided when the truth arrived. Second, the downstream retry loop did what the retry loop always does. EC2's droplet-lease workflow, requeuing work that had timed out, reached the point where “DWFM had entered a state of congestive collapse”. A trace that recorded only "lookup succeeded" would describe none of this.