Production Architecture, Freshness, Monitoring, and Incidents

On 12 June 2025 Cloudflare's Workers KV went down, and every service that depends on it went with it. Cloudflare's post-mortem gives the size of the hole: “This outage lasted 2 hours and 28 minutes, and globally impacted all Cloudflare customers using the affected services.” Nothing in Cloudflare's own ranking, routing or policy code had failed first. The trigger was an outage at a third-party cloud provider.

The other half of the day belongs to Google Cloud, upstream. A quota-policy feature had been added to Service Control on 29 May 2025. Google's incident report is blunt about it: “The issue with this change was that it did not have appropriate error handling nor was it feature flag protected.” On 12 June, blank fields replicated globally within seconds. A null pointer put the binaries into a crash loop worldwide. Recovery in us-central1 took about 2 hours 40 minutes.

This is the shape a recommendation incident actually takes. Read the five findings below against a serving path — ranker, vector index, feature service, eligibility cache, fallback policy. Every one of them transfers.

The online/nearline/batch split is not a teaching device invented for a course. It is how Netflix says its own stack is built. Amatriain and Basilico set the three tiers out in a 2015 handbook chapter, in a section titled “Offline, Nearline, and Online Computation”. Real-time there means a response below a few hundred milliseconds. The example SLA is concrete: recommendations returned in 250 ms for 99% of requests.

That number is what makes the fallback structural rather than a nicety. The chapter says so directly: “Also, a purely online computation may fail to meet its SLA in some circumstances, so it is always important to have a fast fallback mechanism such as reverting to a precomputed result.” The precomputed result exists because the online path is allowed to run out of time.

Freshness and cost then trade along the row. The online path computes request-specific state at serving time: lowest staleness, highest latency sensitivity, strict fallbacks, and the natural home for session and inventory context. The nearline path refreshes embeddings, counters or candidates every few minutes. It balances freshness against cost, and it has to survive events arriving out of order, so its updates have to be idempotent. It carries trends and recent behaviour. The batch tier — what the chapter calls offline computation — rebuilds large aggregates, models and indexes periodically. Efficient, reproducible, able to absorb heavy processing, home to durable profiles and full rebuilds. Stale by design.

The ML code is the small black box in the middle of the picture. A 2015 paper on hidden technical debt in machine learning systems, by ten authors at Google, drew it that way and captioned it: “Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex.” The serving path around that box may include request context, identity, eligibility, user state, candidate services, vector indexes, feature stores, rankers, re-rankers, policy rules, caches, experiments and logging. Reliability is the reliability of all of it together.

The same paper names why component-local health does not compose. Under entanglement it gives the rule a name: “We refer to this here as the CACE principle: Changing Anything Changes Everything.” The scope is broad: inputs, hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection. Change one and the rest move.

Freshness inherits that entanglement. Catalog, inventory, price, rights, user history, embeddings, graph, features, model, index and policy each age at their own rate. One “last updated” timestamp cannot speak for all of them.

Production readiness has a published rubric with numbers on it. Breck and four colleagues at Google wrote it down in 2017 and called it the ML Test Score: 28 specific tests and monitoring needs, across four sections. A test run manually scores half a point; automated, a full point. Then the arithmetic bites: “The final ML Test Score is computed by taking the minimum of the scores aggregated for each of the 4 sections.” You do not average your way out of an untested section. The rubric was calibrated in the field — “We met with 36 teams from across Google” — and it reads a score of 0 as “More of a research project than a productionized system”. Only a score above 5 counts as “Exceptional levels of automated testing and monitoring”.

Run the five operating steps against that rubric. First, define end-to-end SLOs: valid slate rate alongside latency, freshness, coverage and guardrails. Second, version the asset graph — model, feature, index, rules, catalog, creative and experiment — so a rollback can name exact versions. Third, test degraded modes deliberately: source loss, stale data, empty candidates, feature timeouts. Fourth, monitor policy behaviour: source mix, score distributions, exposure, constraints and outcomes. Fifth, prepare rollback and replay: traces, exact versions, working fallbacks, named incident decision rights.

Two of the rubric's monitoring tests are the ones this lesson keeps returning to. Monitor 1 is “Dependency changes result in notification”. The reason given is exactly the failure mode of a shared feature service: “Partial outages, version upgrades, and other changes in the source system can radically change the feature’s meaning and thus confuse the model’s training or inference, without necessarily producing values that are strange enough to trigger other monitoring.” Monitor 4 is “Models are not too stale”. Neither is satisfied by a green dashboard.

Two of the five patterns below have a documented instance attached — a regulator's findings in one case, a vendor's own root-cause report in the other. Both are worth reading before assuming that a serious version disagreement will announce itself.

One request, five things that have to happen inside the latency budget. The budget in the Netflix chapter's example SLA is 250 ms for 99% of requests. Resolve request and identity: surface, session, profile, experiment, privacy mode. Fetch eligibility and context: inventory, policy, rights, current request features. Query candidate services: collaborative, content, graph, trend, exploration. Rank and apply policy: score, constrain, diversify, lay out, and select the fallback behaviour for the moment the budget runs out.

The fifth is not for the user at all. Unless inputs, versions, candidates, scores, slate, exposure and outcomes are traced, nobody can reconstruct tomorrow why this page looked the way it did. Reconstructing 12 June 2025 took two companies and two separate reports.