Monitoring, Feedback, and Change Management

A documentation migration goes through. Latency and error rates stay flat. Complaints rise anyway, because the new parser dropped the effective dates and retrieval is now surfacing obsolete procedures. Service health is necessary and not sufficient. Generative monitoring has to connect infrastructure, component traces, output behavior and downstream outcomes.

Change one part of a machine learning system and you have changed all of it. D. Sculley and nine colleagues at Google named that in 2015, in “Hidden Technical Debt in Machine Learning Systems”, and stated it as a law: “We refer to this here as the CACE principle: Changing Anything Changes Everything.” The principle does not stop at the weights. It reaches “hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak”. A decade later SAE International put the principle in the title of a 2025 technical paper on applying AI and machine learning to automotive safety-critical systems. The acronym outlived the paper because operators keep running into the thing it describes.

A second mechanism sits underneath. Pipelines are underspecified, D’Amour and colleagues argued in 2022: a pipeline is underspecified “when it can return many distinct predictors with equivalently strong test performance”, and those predictors “can behave very differently in deployment domains”. Between the two papers the working assumption follows. A system that was measured once, and a system that is running now, are not the same system.

Dashboards earn their place when an operator can move from symptom to responsible component. Service signals cover availability, queue, latency, throughput, resource use and error codes. Component signals cover retrieval recall proxies, tool errors, context truncation, routing and validator results. Behavior signals cover groundedness, refusal, format, safety, citation and language-quality slices. Outcome signals cover resolution, correction, escalation, complaint, harm, cost and delayed real-world results. The four layers fail independently. The Epic Sepsis Model shows how far apart they can travel.

The vendor's number and an outside number were not close. Wong and colleagues validated the model over 27,697 patients and 38,455 hospitalizations at Michigan Medicine, and reported in JAMA Internal Medicine in 2021: “The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64).” The vendor's reported figure was 0.76-0.83. The model generated alerts on 18% of all hospitalizations. It missed 1,709 of 2,552 sepsis patients, or 67%. It identified 183 patients (7%) that clinicians had not already treated.

An independent replication found the same shape at the outcome layer. Ostermayer and colleagues studied 145,885 encounters across two county emergency departments in 2023 and published the result in JAMIA Open in 2024: sensitivity 14.7%, specificity 95.3%, positive predictive value 7.6%. The median alert lead time was 0 minutes. Nothing in that record is a service outage. The endpoint answered, the alerts fired on schedule, and only the behavior and outcome layers carried the failure.

Each channel reflects something a user did and the way it was gathered. None of them is a neutral sample of what is correct. Explicit ratings — thumbs, stars, written comments — are easy to collect and carry strong selection and emotion bias. Behavioral signals — edits, retries, copy, abandonment, escalation, tool reversal — are available at scale, ambiguous in intent, and can reflect interface friction. Delayed outcomes — later transactions, resolutions, incidents, expert labels — sit closest to real value, arrive slowly or selectively, and require entity and time joins.

Google Flu Trends is the counted case, and its bias came from the platform underneath the signal. From 21 August 2011 to 1 September 2013 the estimates ran too high in 100 weeks out of 108. Lazer and three co-authors reported that in Science in 2014, in the paper they called The Parable of Google Flu. Their figure caption, “GFT overestimation”, gives the size of the gap: “GFT overestimated the prevalence of flu in the 2012–2013 season and overshot the actual level in 2011–2012 by more than 50%.” Part of the drift they attribute to “algorithm dynamics”. Google's own changes to search suggestions and symptom results, in June 2011 and February 2012, altered the input signal while GFT itself sat still.

The same failure mode had been measured a year before that paper. Olson and colleagues, in PLOS Computational Biology in 2013, found that the original model missed the spring 2009 pandemic wave with incidence estimates 30-40 fold too low. The updated model put 2012/2013 epidemic intensity at 268% of ILI surveillance nationally — a 2.68-fold overestimate, 208% regional and 296% local — which the authors summarise as overestimating the season “by 2-3 fold”. A channel that is collected rather than sampled inherits every change made to the thing collecting it.

Track every component that can alter user-visible behavior, and assume the components you did not touch have moved as well.

On 18 January 2024 the chatbot of the UK parcel carrier DPD spent the day insulting its employer. A customer, Ashley Beauchamp, induced it to swear at him and to write a haiku about how terrible the company is. Asked to “exaggerate and be over the top in your hatred”, it replied “DPD is the worst delivery firm in the world”. DPD's public statement the next day, 19 January 2024, named the cause: “An error occurred after a system update yesterday. The AI element was immediately disabled and is currently being updated.” The company added that it had “operated an AI element within the chat successfully for a number of years” before that update. Years of acceptable behavior, one routine update, a public failure the same day. That is why each item below needs its own version and its own rollback path.

Every mutable component should carry a version and a rollback path. Declare the change with its affected components, intended benefit, risks and owner. Run offline gates over representative, critical, adversarial and incident cases. Shadow or canary against production traffic with limited exposure and a comparison. Watch service, component, behavior, safety and cost signals. Join the delayed outcomes — correction, escalation, resolution, incidents, business impact. Then roll forward or back, documenting the decision and preserving the evidence for the next release.

For high-risk systems the last steps are not discretionary. The EU Artificial Intelligence Act, published in the Official Journal of the European Union on 12 July 2024, states in Article 72(1): “Providers shall establish and document a post-market monitoring system in a manner that is proportionate to the nature of the AI technologies and the risks of the high-risk AI system.” Article 72(2) requires that the system “actively and systematically collect, document and analyse relevant data” over the system's lifetime. Article 72(3) requires that it rest on a post-market monitoring plan forming part of the Annex IV technical documentation, with the Commission to adopt a template by 2 February 2026.

Article 73 attaches clocks to the loop's final step. The general case is not later than 15 days from awareness, under Article 73(2). A widespread infringement, or an incident under Article 3(49)(b), gets two days, under Article 73(3). Where a person has died, the limit is 10 days, under Article 73(4). A release record that cannot reconstruct what changed, when it changed and who owned it will not meet a two-day deadline.

Generative applications drift through data, software, policy, model and user change. Effective monitoring detects symptoms, supports diagnosis, and links interventions to later outcomes.

Three regulators wrote that expectation down together. On 27 October 2021 the FDA, Health Canada and the UK MHRA jointly issued ten guiding principles for good machine learning practice in medical device development. The tenth concerns the state after release — “Deployed Models Are Monitored for Performance and Re-training Risks Are Managed” — and reads: “Deployed models have the capability to be monitored in “real world” use with a focus on maintained or improved safety and performance.” It names dataset drift as an example of the degradation to be controlled. Principle 9 requires that users “are also made aware of device modifications and updates from real-world performance monitoring”.

The next lesson prepares for hostile or surprising behavior through red-teaming and incident readiness. Monitoring supplies the evidence needed to contain failures and to decide whether a release should remain live.