ML Security and Supply-Chain Integrity

An attacker never modifies the weights. Instead they alter a feature source, exploit a permissive tool credential, or replace a preprocessing package in the build. The model behaves exactly as trained, on attacker-controlled inputs.

The plainest demonstration of that shape is not an ML incident at all. On 29 March 2024 Andres Freund posted to the oss-security mailing list under the subject "backdoor in upstream xz/liblzma leading to ssh server compromise". He wrote: “The upstream xz repository and the xz tarballs have been backdoored.” Versions 5.6.0 and 5.6.1 of xz/liblzma had been compromised by the project's own upstream release process. The malicious object was extracted from a disguised test file during the build and shipped inside the signed release tarballs, while being absent from the git build path. Anyone reading the source repository saw clean code. Anyone building the published release got the backdoor. CISA issued an alert the same day recommending a downgrade to XZ Utils 5.4.6 Stable. CVE-2024-3094 carries a CVSS v3.1 base score of 10.0 from both Red Hat and NVD, under CWE-506, embedded malicious code.

Nothing in that story required a compromised key, a failed signature, or an altered commit. The release was authentic and the artifact was still hostile. ML security must therefore protect the entire decision path, and the mechanisms that create, approve, distribute and invoke its assets — including the mechanisms whose output is signed, reproducible, and wrong.

Compromise can enter before training, during it, or after the artifact ships. Through evidence inputs: poisoned data, malicious labels, source tampering, hidden trigger patterns. Through build and dependencies: compromised code, packages, images, runners, secrets, provenance. Through artifacts and the registry: unauthorized replacement, alias movement, theft, downgrade. Through the serving interface: evasion, extraction, inversion, injection, denial, abuse. And through operational control: credential misuse, policy bypass, unsafe tools, incident suppression.

The first of those nodes is usually assumed to be the most expensive to attack. It has been priced. A web-scale dataset is distributed as a list of URLs, not as images, so whoever controls a domain controls what the trainer downloads years after the list was compiled. Carlini and eight colleagues attacked that trust assumption by buying expired domains that still served dataset URLs. They report: “By exploiting specific invalid trust assumptions, we show how we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD.” They describe the attacks as immediately practical against 10 popular datasets. Sixty dollars, split-view poisoning, no access to any pipeline.

The intuitive defence against that number is to assume a larger corpus dilutes the poison. That has been measured too, and it does not hold. In October 2025 Anthropic, the UK AI Security Institute and the Alan Turing Institute published a joint study. They pretrained models from 600M to 13B parameters on chinchilla-optimal datasets of 6B to 260B tokens: “We find that 250 poisoned documents similarly compromise models across all model and dataset sizes, despite the largest models training on more than 20 times more clean data.” Those 250 documents are roughly 420k tokens. That is 0.00016% of the largest model's training tokens. Percentage-of-corpus reasoning is the wrong defence model, because the attacker's budget does not scale with the victim's dataset.

A protected builder produces an artifact with valid attestation — and the framework that issues the attestation states in writing what it does not cover.

One security control rarely covers all four. Integrity prevents or detects unauthorized changes to data, code, artifacts, policy and decisions: signing and digests, protected builders, access control and review, behavioral tamper checks. Confidentiality and privacy limit disclosure through artifacts, logs, interfaces and model behavior: least privilege, encryption and minimization, extraction and inversion assessment, controlled telemetry. Availability and abuse resistance maintain service under overload, adversarial queries and resource exhaustion: rate and quota policy, admission control, input bounds, cost-aware fallback. Authority control ensures outputs cannot exceed allowed actions or permissions: tool allowlists, scoped credentials, human approval for consequences, independent policy enforcement.

The confidentiality column is the one that gets priced wrongly, because it is defended as though it were the integrity column — keys, network boundaries, artifact storage. In March 2024 Carlini and colleagues reported what that misses: “For under $20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models.” The attack confirmed hidden dimensions of 1024 and 2048, and the authors estimated under $2,000 to recover the same matrix for gpt-3.5-turbo. There was no access to weights and no breach of storage. The loss ran through the ordinary prediction interface. An access-control review that stops at the network boundary cannot see it. Only an extraction and inversion assessment against the interface as served will.

"Adversarial attack" is too broad for engineering. Ask what the attacker can access, change, observe, or query; what outcome they want; and which controls or costs limit them.

That is not a house convention, and there is a document number for it. NIST's adversarial machine learning taxonomy, NIST AI 100-2e2025, published 24 March 2025, states: “These attacks are classified according to the following dimensions: 1) learning method and stage of the learning process when the attack is mounted, 2) attacker goals and objectives, 3) attacker capabilities, and 4) attacker knowledge of the learning process.” Four dimensions, numbered. A threat model that fills in fewer than four is not yet a threat model. The same taxonomy declines to treat supply-chain compromise as an afterthought to the model-level attacks. It carries its own identifier: NISTAML.05.

A public prediction API creates different risks from an internal batch model, an on-device biometric system, or a tool-using language model. Fill in stage, goal, capability and knowledge for the interface you actually ship, and the control list stops being generic. A $20 query budget, a $60 domain purchase and a compromised release tarball land in three different cells.

Use a concrete attacker and asset graph rather than a generic checklist. Inventory assets and authority: data, models, prompts, tools, secrets, routes, and external actions. Define attacker capabilities along the four NIST AI 100-2e2025 dimensions — stage of the learning process, goals and objectives, capabilities, knowledge. Trace attack paths from each entry point to the decision and consequence it reaches. Place independent controls: isolation, signing, least privilege, validation, quotas, monitoring. Then test and rehearse, with adversarial evaluation, dependency-compromise and credential-abuse drills, and an incident response run end to end.

Four concrete questions give those steps something to bite on, and each has a published price attached. Could a $60 expired-domain purchase reach anything in your training or fine-tuning corpus, and would you know? Would 250 documents inserted upstream be caught by a filter that reasons in percentages of the corpus? Would an extraction attempt running at under $20 through your served interface be visible in your telemetry, or does it look like a paying customer? And if your build were the xz build — reproducible, signed by the maintainer, and hostile — which control on your list catches it before the artifact ships?

Signed provenance, vulnerability results, data-access review, adversarial tests, runtime policy, and incident readiness can all travel with a release. Each artifact supports one specific claim and no more. The component inventory is the clearest case. NIST SP 800-204D, from February 2024, states the limit: “while SBOMs enable the identification of components and provenance, they do not provide enough information to address vulnerabilities nor content to address software defects. Hence, SBOMs alone cannot be used for vulnerability management.” An inventory is real evidence of what went into the build. It is not a verdict on whether what went into the build is safe — the same limit the SLSA specification states about its own levels.

Avoid the sentence "the model is secure." State which attack paths were tested, under what assumptions, and which risks remain. That the dependency set was policy-checked but the upstream release process was trusted. That extraction was rate-limited but not proven infeasible. That the injection classifier reduces attempts, while authorization — not the classifier — bounds what a successful one can do.