Production Assignment, Drift, and the Cluster Lifecycle

Operational monitoring must inspect profiles and consequences, not only label counts. None of the failures below shows up as a change in the number of rows per cluster. At least one of them already has a name. A 2021 systematic mapping study of AI-based systems read 21 primary studies and catalogued 72 antipatterns with 46 proposed solutions. The list a team monitors against does not have to be invented from scratch.

The assignment mechanism has to match the fitted model and come with a rejection policy. Whether a fitted clustering can assign a new observation at all is not a design choice. It is a documented property of the algorithm. scikit-learn's user guide classifies K-Means, mean shift, affinity propagation, Gaussian mixtures, BIRCH and bisecting K-Means as inductive, and spectral clustering, Ward, agglomerative clustering, DBSCAN, HDBSCAN and OPTICS as transductive. The consequence is stated in one line: “Transductive clustering methods (in contrast to inductive clustering methods) are not designed to be applied to new, unseen data.” That sentence is why the surrogate-classifier column exists at all.

Where a library does offer prediction for a transductive method, it says what it is doing rather than pretending the limitation is gone. The hdbscan documentation describes HDBSCAN* as transductive and supplies approximate_predict. That function holds the existing clustering fixed. It returns a label — including -1 for noise — with a prediction strength for each new point. Nothing is re-fitted. The new observation is scored against a frozen structure, and it is allowed to come back as noise. That is the contract a production assignment service has to state explicitly, whichever of the four mechanisms below it is built on.

Google Flu Trends went on producing estimates long after they stopped being right, and the count is on the record. A 2014 paper in Science set the published estimates against the surveillance data and recorded the run: “From 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence 100 out of 108 weeks.” In the 2011–2012 season the model overshot by more than 50%. The output stream itself announced nothing. A number arrived every week, on schedule, in the shape it had always had. The system was in the end retired rather than repaired: on 20 August 2015 Google announced it would stop maintaining the Flu Trends site, handing the raw signal to CDC, Columbia and Boston Children's Hospital instead.

A clustering deployment fails the same way. Production assignment is not exploratory clustering with a scheduler attached. A deployed system needs rules for unknowns, version changes, historical labels, and the actions tied to each group. It also needs a way of noticing when the labels have stopped describing anything. A steady label stream is exactly what a stale model produces.

Treat assignment, rejection, consumer migration, and monitoring as one system design: freeze the artifact set, define unknown behavior, instrument every assignment, monitor several drifts at once, compare candidate refreshes against the incumbent with a rollback prepared, and retire deliberately.

Step 4 is not this lesson's private recommendation. On 27 October 2021 the FDA, Health Canada and the MHRA jointly issued ten guiding principles for good machine learning practice in medical device development. Number 10 is titled “Deployed Models Are Monitored for Performance and Re-training Risks Are Managed”. It names dataset drift explicitly among the degradation risks that require controls once a model is retrained after deployment. Three regulators put their names to a numbered principle that treats monitoring and retraining risk as a single subject. That is the argument for tracking label mix, feature profiles, boundary margins, unknown rate and action performance together, rather than watching the first of the five and calling it observability.

Somebody has to govern a cluster from the day its evidence is gathered to the day it is retired: design, launch, operate, refresh, migrate, retire.

The last stage is the one treated as paperwork, and Zillow paid for it in public. On 2 November 2021 the company announced it was winding down Zillow Offers because its home-price model could no longer be trusted. The Q3 2021 results release disclosed a write-down of inventory of approximately $304 million in the Homes segment and a further $240–265 million of expected Q4 losses. The wind-down cut roughly 25% of a workforce of more than 6,400. The quarter shows what an automated valuation running past its evidence does to the operation built on top of it: 9,680 homes bought, 3,032 sold. Zillow Group's co-founder and CEO, Rich Barton, gave the reason in the release itself — “We've determined the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility”. Retirement was not a cleanup task at the end of a checklist. It was the decision, and it arrived with a balance sheet.

Cluster labels end up inside APIs, dashboards, decisions, and user experiences. That makes versioning and migration as important as the clustering algorithm. The system should admit novel observations and expose uncertain assignments. It should retire obsolete groups. Stable label counts alone are not evidence that the structure remains useful.

For one class of system this has stopped being advice. The EU's Artificial Intelligence Act, adopted in 2024, makes monitoring a duty of the provider. Article 72(1) states that “Providers shall establish and document a post-market monitoring system in a manner that is proportionate to the nature of the AI technologies and the risks of the high-risk AI system.” Article 72(2) requires that system to collect and analyse performance data “throughout their lifetime”, and the monitoring plan itself forms part of the technical documentation under Annex IV. The obligation runs for as long as the system is in use, not until launch. A team that cannot say which cluster version produced a label, what it was used for, or how the population under it has moved does not have a documentation gap. In that jurisdiction, for that class of system, it has a compliance one.