Observability for ML Systems

Every metric gets a request-level identifier attached. What happens next is arithmetic, and the vendor publishes it. Google Cloud Monitoring states that total cardinality is the product of metric and resource cardinality: “If you have a metric with cardinality of 1,000, and a resource with cardinality of 100, and if every label value appears, then you have 100,000 time series (1,000×100)”. A request ID is not a dimension of cardinality 1,000. It is a new label value for every call. The multiplication has no upper factor at all.

Two independent organisations publish the rule against it, an open-source project and a commercial vendor, in almost the same words. Prometheus puts it in a CAUTION note: “Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.”

Neither of them is describing an edge case. They are describing what a request ID does to a metric store.

A prediction service emits thousands of metrics with model ID, customer ID, feature name, and request ID as labels. The monitoring backend becomes expensive and slow. Operators still cannot connect a bad decision to the feature values and policy used.

Observability is the ability to ask useful questions about internal state from external evidence. It requires correlation, semantics, and bounded telemetry—not logging everything.

Most of an ML system is not the model, and that got a number at the start of the MLOps literature. Sculley and nine co-authors published “Hidden Technical Debt in Machine Learning Systems” in 2015. A mature system, they estimated, “might end up being (at most) 5% machine learning code and (at least) 95% glue code”. Their caption to Figure 1 says it as a picture: “Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex.”

The same paper names monitoring itself as a system-level debt, and its warning is the one this lesson exists to answer: “Even monitoring ML system behavior may prove difficult without careful design”. If at most five percent of the system is the model, telemetry aimed only at the model is aimed at the wrong ninety-five percent.

The most useful detail depends on the question and the time horizon. Hot operational evidence is queried in minutes and thrown away. Warm diagnostic evidence is sampled, access-controlled, and kept long enough to reconstruct a cohort. Cold audit evidence is a narrow, deliberate schema, kept because someone may contest a decision.

For the cold tier, “retention follows policy” now has article numbers behind it. The EU Artificial Intelligence Act — Regulation (EU) 2024/1689, of 13 June 2024 — makes the record compulsory for high-risk systems: “High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system.” (Article 12(1)). Article 19(1) sets the floor for keeping them. Providers must retain the automatically generated logs “for a period appropriate to the intended purpose of the high-risk AI system, of at least six months”, unless other Union or national law provides otherwise.

Six months is not a storage-budget decision for a high-risk system. It is the statutory minimum. The schema that survives that long is the one an operator will have to defend.

A distributed trace can show the path through authentication, feature lookup, inference, and policy. Metrics summarize rates and distributions. Logs preserve discrete events and context. A decision envelope links these signals to the exact model, feature set, threshold, fallback, and output.

No single signal is sufficient. Correlation identifiers and context propagation allow operators to move from an alert to a trace, then to structured records and affected cohorts.

Correlation has a standard behind it. W3C made Trace Context a Recommendation on 23 November 2021. It defines a traceparent header carrying a version, a trace-id, a parent-id and trace flags. The trace-id is “a 16-byte array”; the parent-id an “8 bytes array identifier”. OpenTelemetry requires the same header and the same field sizes.

The step from a bounded metric down to one individual request is specified too. OpenTelemetry's metrics data model defines an exemplar as “a recorded value that associates OpenTelemetry context to a metric event within a Metric”, one component of which is the optional trace association given as (trace_id, span_id). That pair is the whole bridge. The metric keeps a handful of dimensions. A few sampled points carry the identifiers that open the trace, the logs and the decision record for the call behind the spike.

Collect each signal for the question it answers best. Metrics say how often, how much, and how a distribution moved. Traces say where time and failure went in one request. Logs and events preserve the discrete transition. Decision evidence binds model, features, policy, fallback and outcome together.

The last of those is the one teams discover last and need most, and it was already on the list in 2015. Sculley et al. offered prediction bias, action limits and up-stream producers as starting points for what to monitor. Not model accuracy. The places where a model's behaviour touches the surrounding system that makes up the other ninety-five percent.

Begin with what an operator must decide, then choose the minimum evidence: list the diagnostic questions, assign each one a signal, define the correlation identifiers that connect them, bound detail and retention, then test the investigation.

Step four has measurements behind it rather than instinct. Dapper priced the choice: 16.3% added average latency at 1/1 sampling against −0.20% at 1/1024, with spans averaging 426 bytes. The sampling rate you pick is a figure you can defend in a design review instead of a preference.

Step five is published doctrine at two large operators. Google's The Site Reliability Workbook (2018) describes company-wide Disaster Recovery Testing (DiRT) alongside smaller Wheel of Misfortune exercises, and states the rule for them: “Teams respond to the controlled emergency as if it were a real emergency.” AWS numbers the same practice. REL12-BP05 of the Well-Architected Framework Reliability Pillar, “Conduct game days regularly”, instructs: “To help you reproduce real-world scenarios where system components or services may fail unexpectedly, inject simulated faults as a game day exercise.” The exercise runs in a replica environment. Teams observe system behaviour with CloudWatch and X-Ray while the fault is live, and the same document's REL06-BP07 requires end-to-end tracing of requests through the system — the thing the game day is watching. The drill tests the telemetry, not only the responders.

Dashboards should implement a diagnostic model that the team already understands. Adding charts without a question creates noise and encourages metric ritual.

Review telemetry after incidents. Which evidence arrived too late, which fields were unnecessary, and which correlation broke?

Google's SRE book offers a default for that diagnostic model: “The four golden signals of monitoring are latency, traffic, errors, and saturation”. The advice that follows is blunt: “If you can only measure four metrics of your user-facing system, focus on these four.” Four is a smaller number than the thousands of labelled series the case section began with. It is also the number a team can actually reason from during an incident.