Recommender Lifecycle, Change Management, and Retirement

Knight Capital stopped using a function called "Power Peg" in 2003. Nobody removed the code from the production order router. Nine years later it ran. The SEC's order of 16 October 2013 puts the whole of this lesson in one sentence: “Despite the lack of use, the Power Peg functionality remained present and callable at the time of the RLP deployment.”

The mechanism was ordinary. The replacement RLP code went out in stages from 27 July 2012, ahead of a 1 August 2012 launch. It repurposed the very flag that had once activated Power Peg. One of Knight's eight SMARS servers never received it. On 1 August 2012 that server met 212 incoming parent orders with the old code. Out came 4 million executions in 154 stocks, more than 397 million shares in approximately 45 minutes, and a $460 million loss. Knight paid a $12,000,000 civil penalty.

A recommender fails the same way, and more quietly. The ranker is switched off. Its item embeddings, candidate cache and source quota stay live. The retired policy goes on shaping what users see, with no owner and nobody checking.

Lifecycle management covers proposal, design review, data and objective approval, offline evidence, shadowing, experimentation, rollout, monitoring, incident response, retraining, migration, and retirement. A model can be undeployed while derived embeddings, profiles, indexes, rules, or cached slates carry on working.

Bank supervisors wrote this into regulation long before recommender teams learned it. SR 11-7, issued on 4 April 2011, instructed banking organizations: “Also, organizations should maintain an inventory of models implemented for use, under development for implementation, or recently retired.” A retired model stayed on the books. Turning it off did not close the file on it.

That guidance stood for fifteen years. On 17 April 2026 the Federal Reserve, OCC and FDIC superseded it with revised interagency guidance, SR 26-2. Its Model Inventory section covers “models under development or in use”, and no longer names recently retired models. The inventory rule changed. Retirement did not. It still has to follow the complete asset and dependency graph, one edge at a time.

Version drift accumulates while the system runs. Evidence destruction bites long after it stops. Model, index, rules and features advance independently without compatibility checks; then the artifacts an investigation would need are deleted before the obligations expire. Every item below has already appeared in this lesson wearing someone else's uniform — a trading firm's order router, a photo app's face embeddings, a search company's undeclared readers.

Proposal and contract at one end, retire and retain at the other, with build and validate, progressive release, and operate and change between them. Most teams have the middle three. They discover at the far end that removing a system's influence is harder than shipping it was.

Progressive release is the stage where the base rate bites, and the base rate is published. Most ideas fail. “Only one third of the ideas tested at Microsoft improved the metric(s) they were designed to improve”, Kohavi and colleagues reported in 2013. They cited comparable rates elsewhere: about 10 percent of Google's controlled experiments led to business changes, and Netflix considers 90 percent of what it tries to be wrong.

So shadow, canary, experiment, guardrails and rollback are not ceremony around a change that was probably good. They are cheap machinery for discovering that most changes are not. Which is also why each of them needs a stop threshold written down before the run begins.

An inventory of dependencies opens the retirement sequence: services, indexes, caches, experiments, reports and downstream consumers. Then migration is defined — compatibility, dual-run, shadow and fallback periods. Then evidence is preserved, influence removed, and closure verified by testing that no route, job or dashboard still reaches a retired asset.

Steps three and four are the ones teams collapse into each other. A regulator has already written out what keeping them apart looks like. The FTC's order of 6 May 2021 against Everalbum set three separate clocks. One ran 30 days, for photos and videos of deactivated users. One ran 90 days, for face embeddings created without express consent. A third ran 90 days for “Affected Work Product,” defined as any models or algorithms developed in whole or in part using biometric information collected from Ever users. The order is explicit about the form closure takes: “Within ninety (90) days after the issuance of this Order, delete or destroy any Affected Work Product, and provide a written statement to the Commission, sworn under penalty of perjury, confirming such deletion or destruction.”

Note what the order does not do. It expressly permits retention of that same material where required by law, court order or the rules on safeguarding evidence in pending litigation. Preservation and destruction run on different clocks against the same assets. And the derived artefacts — the embeddings, the trained model — carry deadlines of their own. They do not disappear when the source data does.

A model upgrade can alter score distribution, candidate mix, calibration, latency, provider exposure, and user controls. Release evidence should cover each affected layer. Retirement tests can query historical item IDs, old profile features, and source tags to confirm that the policy no longer influences recommendations. That is the recommender's version of asking whether the old function is still callable, rather than whether anyone still calls it. Keep a named owner until verification is complete.