Skip to content
AI.info

MLOps

Registries, Approval, and Release Candidates

Use registries and approval workflows to control immutable candidates, mutable aliases, lifecycle states, and release conditions.

By the end you can

A registry is an index, not a quality oracle

Registering a model makes it discoverable and gives it an identity. It does not prove that the candidate meets the production contract, or that its dependencies are still compatible.

A useful registry sits inside a release process. It stores or references immutable candidates, metadata, evidence, ownership, lifecycle state, and the mutable aliases used for routing.

Visual

Lifecycle states describe readiness, not identity

A candidate can move through review while its immutable version stays fixed.

FigureTimeline · 6 stops
  1. Draft

    The bundle is incomplete or still changing.

  2. Candidate

    Required assets exist and automated gates have passed.

  3. Approved

    Accountable reviewers accepted the declared conditions.

  4. Active

    Traffic or scheduled work can resolve to this version.

  5. Deprecated

    The version remains accessible but should be replaced.

  6. Retired

    Execution is blocked while evidence may be retained.

Example

The alias moved, but the audit record did not

A tag is not a version. NIST's container security guide tells organisations to access images “using immutable names that specify discrete versions”. Under the heading “Stale images in registries” it explains why. A “latest” tag “is only a label attached to the image and not a guarantee of freshness”. That guidance is from September 2017. This failure is documented industry guidance, not a hypothetical.

Kubernetes describes the same mechanics from the other end. A registry can repoint a tag. Some Pods then run the old code and some run the new one, while every deployment manifest still reads the same. The fix it names is the digest. “An image digest uniquely identifies a specific version of the image, so Kubernetes runs the same code every time it starts a container with that image name and digest specified,” the Kubernetes documentation says. A model alias behaves exactly like a tag. A resolved model version behaves exactly like a digest.

  • Convenient deployment: Serving code resolves a mutable tag or alias at start-up, so each replica picks up whatever the registry currently points it at.
  • Missing resolution: Request logs store the mutable name, not the immutable target it resolved to — in container terms, the tag rather than the `<image-name>@sha256:…` digest.
  • Later promotion: The registry repoints the tag at a new image. That is precisely the stale-reference case NIST SP 800-190 raises.
  • Broken investigation: Kubernetes gives the observable result — some Pods on old code, some on new — while the historical logs show one unchanging name and imply one version.
  • Correct repair: Pin by digest and resolve the alias at load or request time, then persist that exact target so the record names the code that actually ran.

The candidate should include everything needed to decide

A release candidate is stronger than a weight file. It should bind the runtime artifact, input and output contracts, transformation dependencies, evaluation report, known limitations, operational profile, and rollback target.

A regulator has already written down what such a bundle must contain when the model is expected to change. In the US, a device manufacturer can pre-authorise future model changes by getting a Predetermined Change Control Plan approved. The FDA's guidance defines it: “Predetermined Change Control Plan (PCCP): The documentation describing what modifications will be made to a device and how the modifications will be assessed.” A PCCP has to bind three things, and a metric table is not one of them: a Description of Modifications, a Modification Protocol — the verification and validation activities, with acceptance criteria fixed in advance — and an Impact Assessment. The FDA issued that guidance on 4 December 2024 and reissued it on 18 August 2025. The statute behind it, section 515C of the FD&C Act, dates from 29 December 2022.

The registry can index that bundle. Approval should evaluate the product contract. A candidate can be technically valid and still fail capacity, fairness, security, or workflow requirements.

Case

The document that reports where a model should not be used

A model's evaluation is easy to publish. Where the model should not be used is the part that goes missing. Mitchell and eight co-authors proposed a standard container for both in 2019, in a paper called Model Cards for Model Reporting.

Model cards, the paper says, are “short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups … and intersectional groups”. They also “disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information”. That last part is the one an approval needs. A metric table never supplies it.

Comparison

Automated gates and human approvals protect different boundaries

Neither should be used as theatre, and a passing gate is not a substitute for the third column.

CrowdStrike's Channel File 291 update of 19 July 2024 is the demonstration. A Rapid Response Content bundle went straight to the whole fleet. The IPC Template Type defined 21 input parameter fields. The integration code that invoked the Content Interpreter supplied 20. The Content Validator passed the bundle anyway, because it too assumed 21 inputs, and the resulting out-of-bounds read crashed Windows hosts. Microsoft estimated 8.5 million Windows devices affected, less than one percent of all Windows machines. The automated gate ran and cleared the release. What was absent was traffic control.

The sixth finding of CrowdStrike's own root cause analysis, published on 6 August 2024, is titled “Template Instances should have staged deployment”. The mitigation added afterwards reads: “New Template Instances that have passed canary testing are to be successively promoted to wider deployment rings or rolled back if problems are detected.” Canary testing, deployment rings and bake-in time were not exotic requirements. They were the column this release skipped.

FigureComparison · 3 columns

Automated gate

Checks repeatable conditions at machine speed.

  • Schema and artifact integrity
  • Metric and slice thresholds
  • Security scans and policy checks
  • Blocks promotion on deterministic evidence

Accountable approval

Evaluates context, exceptions, and residual uncertainty.

  • Reviews intended use and harms
  • Considers operational readiness
  • Accepts explicit conditions
  • Records who made the decision

Traffic control

Moves a tested candidate into exposure gradually.

  • Uses aliases or routing weights
  • Supports canary and shadow modes
  • Preserves rollback target
  • Produces deployment evidence

Key idea

Approval can become a rubber stamp

A long checklist does not create meaningful governance when reviewers cannot inspect the evidence or challenge the release. That claim has been measured. DORA's 2019 Accelerate State of DevOps Report found that formal change management requiring approval by an external body — a change advisory board, or a senior manager — had a negative impact on software delivery performance. Under the heading “Heavyweight change process” it reports the size of the effect: “Survey respondents were 2.6 times more likely to be low performers if their organization had this kind of formal approval process in place.” It found no evidence that a more formal approval process was associated with lower change fail rates.

What it recommends instead is segregation of duties implemented through peer review: requiring “every change be approved by someone else on the team as part of code review”, with authenticated approval events logged. The earlier 2014 State of DevOps Report, with more than 9,200 respondents, had already reported the same direction of effect. So separation of duties should follow conflicts of interest and decision authority, not arbitrary handoffs.

Review quality can be measured through rejected candidates, requested changes, time spent on high-risk evidence, and post-release findings — not approval volume alone.

A gate is effective only when it can stop or condition the release.

Case

One server missed the deployment, and Knight lost $460 million

Knight Capital lost a firm to a partial deployment. The SEC's order of 16 October 2013 records the cause in one line: “one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers”. It also records that “Knight did not have a second technician review this deployment” and “had no written procedures that required such a review”.

The new code “repurposed a flag that was formerly used to activate the Power Peg code”. That older code was still present, and still callable, on the eighth server. On 1 August 2012 the 212 parent orders processed by the defective code produced “4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes”. After that, “Knight lost over $460 million from these unwanted positions”. Seven of eight servers had the release. One did not. No gate and no reviewer stood between that fact and the market.

Figure

One server of eight missed the release, and the forty-five minutes that followed, counted out from the SEC's order.

Steps

Promote an immutable candidate safely

The workflow should preserve exact identity while allowing controlled routing: assemble the bundle, run automated gates, perform accountable review, assign a mutable route, and record the resolution.

Step 3 is where the scope of the approval gets written down, and in the EU that scope has a legal edge. The AI Act — Regulation (EU) 2024/1689, adopted on 13 June 2024 and published in the Official Journal on 12 July 2024 — requires in Article 43(4) that a high-risk AI system already assessed go through a new conformity assessment procedure in the event of a substantial modification. There is one carve-out, and it is the interesting part: “For high-risk AI systems that continue to learn after being placed on the market or put into service, changes to the high-risk AI system and its performance that have been pre-determined by the provider at the moment of the initial conformity assessment and are part of the information contained in the technical documentation referred to in point 2(f) of Annex IV, shall not constitute a substantial modification.”

Article 3(23) defines 'substantial modification' as a change “which is not foreseen or planned in the initial conformity assessment carried out by the provider”. That is what step 3 is recording: the envelope inside which later versions may be routed, and the boundary past which the approval is void. Registration belongs to the same discipline. Article 49(1) requires the provider, or where applicable the authorised representative, of a high-risk AI system listed in Annex III — other than those in point 2 of Annex III — to register themselves and their system in the EU database referred to in Article 71 before it is placed on the market or put into service. Step 5 then persists the exact version each deployment and decision resolved to.

FigureProcess · 5 steps
  1. 1. Assemble the bundle

    Bind artifacts, contracts, evidence, limitations, and rollback target.

  2. 2. Run automated gates

    Validate integrity, compatibility, metrics, security, and policy.

  3. 3. Perform accountable review

    Record approval, conditions, exceptions, and effective scope.

  4. 4. Assign a mutable route

    Move an alias or traffic policy to the approved immutable version.

  5. 5. Record resolution

    Persist the exact version used by each deployment and decision.

The registry should make the safe path easier

A registry succeeds when engineers can find approved assets, compare evidence, reproduce how a candidate was chosen, and roll back without private knowledge. It fails when it becomes a catalog of orphaned experiments.

That is a codified practice rather than a preference. NIST SP 800-218, the Secure Software Development Framework, version 1.1 of February 2022, states it as a numbered task: “PS.3.1: Securely archive the necessary files and supporting data (e.g., integrity verification information, provenance data) to be retained for each software release.” PS.3.2 asks for collecting, safeguarding, maintaining and sharing provenance data for all components of each release — in an SBOM, for example. PS.2.1 requires making integrity verification information available.

The framework maps those tasks to Executive Order 14028 of 12 May 2021. Its section 4(e) requires “providing a purchaser a Software Bill of Materials (SBOM) for each product”, and requires “attesting to conformity with secure software development practices”. Retention and retirement rules should therefore be designed early. Otherwise the registry stops being an operational control and becomes an archive nobody trusts.

Key takeaways