Versioning the ML Asset Graph

A registry reports that model version 42 is live. The answer still omits the feature pipeline, category vocabulary, request schema, threshold policy, runtime image, and retrieval index used by the service.

Knight Capital Americas LLC could have answered the registry question correctly on every server it owned and still lost $460 million in about 45 minutes. A staged deployment beginning 27 July 2012 was meant to put the new RLP code on eight SMARS servers. One server did not get it. The SEC's order of 16 October 2013 records what happened: “During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers.” A flag the new release had repurposed then re-activated the discontinued “Power Peg” code on that eighth server. On 1 August 2012, 212 incoming parent orders became roughly 4 million executions in 154 stocks for more than 397 million shares in about 45 minutes. The penalty was $12,000,000.

Parties with nothing to do with the settlement read the episode the same way. Carol Clark, writing for the Federal Reserve Bank of Chicago, reported the 1 August 2012 date, the 45-minute window, a near-bankruptcy and a $440 million loss caused by “a software malfunction”. A WilmerHale client alert put the cause at “a router that contained discontinued and therefore incorrect code”.

Seven servers ran the release. One ran something else, and no name in any registry distinguished them. An ML release is a connected asset graph. Versioning succeeds when a production decision can be traced through that graph without relying on mutable names or personal memory.

The Knight order describes a missing copy of code. The same shape of failure appears in machine learning when an edge is missing from the release record rather than a file from a server. Work through a text classifier that passes its offline test and fails after deployment.

The model artifact is one node in a larger compatibility graph.

Each class below is a class of node. A release is valid only when the edges between the classes are declared rather than assumed. The division is not a taxonomy for its own sake. Evidence, transformation, learned, execution and decision assets each fail differently, and each of them fails quietly when the release record omits its edge.

An immutable identifier answers which exact object was used. A mutable alias answers which object a role currently points to. Both are useful. They must not be confused.

A deployment can target an alias such as `champion`; the decision record should resolve and store the exact version at the moment the deployment runs. Otherwise what the team can rebuild from history changes whenever the alias moves.

Name-and-version is itself a reference, not an identity, and there is a timestamped incident to prove it. On Tuesday 22 March 2016, after a name dispute over the kik package, Azer Koçulu unpublished 273 packages from the npm public registry. Isaac Z. Schlueter wrote on the npm blog the next day: “In this case, though, without warning to developers of dependent projects, Azer unpublished his kik package and 272 other packages.” One of those 272 was left-pad. From shortly after 2:30 p.m. Pacific, npm observed hundreds of dependency-resolution failures per minute. A community replacement published as 1.0.0 did not help. Dependency chains including babel and atom explicitly requested 0.0.3, and for two and a half hours that string resolved to nothing at all. npm ended the outage only by restoring the original left-pad 0.0.3 from backup at 4:55 p.m.: “The duration of the disruption was 2.5 hours.” Chris Williams reported the same event for The Register that day.

A build that names a remote dependency by version is reproducible only for as long as some other party keeps serving that exact object under that exact name. Content identity — a digest resolved and stored at release time — is the part of the record that no third party can withdraw.

A clean release process uses each concept deliberately.

The immutable column is not a local convention. Semantic Versioning 2.0.0 states it as a rule: “Once a versioned package has been released, the contents of that version MUST NOT be modified. Any modifications MUST be released as a new version.” That MUST NOT is normative rather than advisory, because the document adopts the RFC 2119 keywords. The same specification requires MAJOR to be incremented for backward incompatible changes to the public API.

The npm public registry enforces the same property operationally, across millions of packages. Its Unpublish Policy states: “Registry data is immutable, meaning once published, a package cannot change... So if you've ever published a package called "bob" at version 1.1.0, no other package can ever be published with that name at that version. This is true even if that package is unpublished.” Unpublishing at all is bounded by the policy's 72-hour and 300-downloads-per-week conditions.

A mutable alias makes no such promise, and is not supposed to. That is the whole distinction. One column is a rule about contents. The other is a name whose target is expected to move.

Draw the graph before anything is promoted.

Step 1 does not have to be invented, because a published minimum already exists. Executive Order 14028, published in the Federal Register on 17 May 2021, defines an SBOM, and the supply-chain guidance it ordered has to cover “providing a purchaser a Software Bill of Materials (SBOM) for each product directly or by publishing it on a public website”. NTIA set the floor on 12 July 2021, at exactly seven per-component data fields: Supplier Name, Component Name, Version of the Component, Other Unique Identifiers, Dependency Relationship, Author of SBOM Data and Timestamp, alongside Automation Support and Practices and Processes. The report's “Data Fields” section gives the reason: “The goal of these fields is to enable sufficient identification of these components to track them across the software supply chain and map them to other beneficial sources of data, such as vulnerability databases or license databases.”

Supplier, name and version are the identity triple. The report defines the third as an “Identifier used by the supplier to specify a change in software from a previously identified version”. A release that cannot fill those three fields for its tokenizer vocabulary, its calibration object or its runtime image has not finished step 1, whatever the registry says about the model. Dependency Relationship is the field that carries the compatibility edges of step 3. Timestamp and Author of SBOM Data are what make the resolved aliases of step 4 evidence rather than assertion.

The strongest versioning test begins with a real production output. Can the team identify the input envelope, feature values, transformation versions, model, policy, runtime, and fallback that produced it?

If the answer relies on what an alias points to today, the record is not historical evidence. The SEC could say which of eight servers held which code, because the deployment left a trace that survived the deployment. A build that asked for left-pad 0.0.3 could say what it wanted but not what it had. A decision record is the artifact that closes that gap for one output at one moment. It closes it only if every mutable reference in it was resolved when the decision was made.