Data Security, Poisoning, and Supply-Chain Risk

A marketplace fraud model retrains weekly from transactions, seller metadata, and investigator outcomes. Nothing in the list below is exotic. Nothing on it is hypothetical either — every move has a dated, named counterpart later in this lesson. A dependency compromised on a public index: torchtriton, December 2022. A corpus whose contents changed under a stable download URL: split-view poisoning of LAION-400M. Contaminated records found by an outside auditor and recalled: Re-LAION-5B, August 2024. Malicious code shipped inside a release tarball: CVE-2024-3094. The pipeline accepted every one of them.

A threat model names assets, actors, capabilities, trust boundaries, entry points, and consequences. For ML data the asset is not only confidentiality. It is integrity, provenance, availability, and label independence too. Attackers may seek to exfiltrate sensitive records, bias a model, create a targeted backdoor, hide their behavior, exhaust pipeline capacity, or corrupt evaluation evidence. Controls should follow the path from collection through storage, transformation, annotation, release, serving, retention, and deletion.

That vocabulary is no longer only an engineering convention. Resilience to these attacks is a legal requirement for high-risk AI systems in the EU, and the statute enumerates the threat classes rather than gesturing at them. Article 15(5) of the AI Act, Regulation (EU) 2024/1689: “The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent, detect, respond to, resolve and control for attacks trying to manipulate the training data set (data poisoning), or pre-trained components used in training (model poisoning), inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws.”

Five classes, one statute: data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks, and model flaws. Recital 76 of the same regulation adds that cyberattacks on AI can leverage training data sets or trained models as AI-specific assets. The data and the weights are the target, not merely the container around them. A threat model whose asset list stops at unauthorized reads is missing four of the five classes the regulation names.

Different stages give an attacker different things to tamper with and different things to read. Collect, Transport, Store, Transform, Label, and Release each fail in their own way. The Transform stage's entry marked “dependency compromise” has a dated name attached to it.

Between 25 and 30 December 2022, PyTorch-nightly's Linux pip packages pulled a malicious torchtriton uploaded to PyPI, which took precedence over PyTorch's own package index. PyTorch disclosed it on 31 December 2022: “PyTorch-nightly Linux packages installed via pip during that time installed a dependency, torchtriton, which was compromised on the Python Package Index (PyPI) code repository and ran a malicious binary.” That binary read /etc/hosts, /etc/passwd, $HOME/.gitconfig, $HOME/.ssh/* and the first 1,000 files in $HOME, and exfiltrated them over encrypted DNS to *.h4ck[.]cfd. Its SHA256 was 2385b29489cd9e35f92c072780f903ae2e517ed422eae67246ae50a5cc738a0e. PyTorch renamed the dependency to pytorch-triton and registered a placeholder on PyPI.

Notice which boundary actually failed. Not PyTorch's code, which was unchanged. Not the build system, which worked. Not the credentials, which were valid. The trust boundary was the resolver's choice of which index to believe, and it was one nobody had drawn on a diagram. A training host is a machine holding SSH keys and a git configuration. A dependency name is an instruction to fetch and execute code on it.

A resilient design uses several control classes. No single control stops every attack. Six national agencies have published a concrete list for this exact threat, and it is worth reading against the four classes rather than instead of them.

On 22 May 2025, six agencies — NSA AISC, CISA, FBI, ASD's ACSC, NCSC-NZ and NCSC-UK — issued a joint Cybersecurity Information Sheet on securing the data used to train and operate AI systems. It organises AI data risk into three areas: data supply chain, maliciously modified (“poisoned”) data, and data drift. Its named mitigations are four. Raw-data cryptographic hashes attached by the curator. Hash verification at download, with discard on mismatch. Periodic re-scraping. Curator certification at publication. Those map onto the control classes without much strain. The curator's hashes and the certification prevent silent substitution and detect it after the fact. Discard-on-mismatch limits what a swapped file can reach. Periodic re-scraping is how you rebuild from a known-good input rather than trusting the copy you already hold.

The sheet also prices the attack those four controls defend against. It puts split-view poisoning of a curated set such as LAION-2B or COYO-700M at roughly $1,000, and as little as $60. Frontrunning poisoning reaches further: “Industry analysis demonstrated potential malicious actors would be able to successfully poison as much as 6.5% of Wikipedia.” A control budget is being compared against an attack budget of three or four figures.

The procedure should make it hard to swap in an unauthorized file, and easy to notice when someone does. Two of its steps — pin dependencies, sign the manifest — already exist in an official form, with the fields spelled out.

Both are required of federal suppliers. Executive Order 14028, “Improving the Nation's Cybersecurity”, signed 12 May 2021, requires them to maintain provenance data for code and components and to provide a Software Bill of Materials for each product. The order also defines that bill, in the line this lesson's analogy section is built on: “It is analogous to a list of ingredients on food packaging.”

What has to be on the label was fixed two months later. On 12 July 2021 NTIA set the baseline at seven data fields: Supplier, Component Name, Version of the Component, Other Unique Identifiers, Dependency Relationship, Author of SBOM Data, and Timestamp. It adds automation support — SPDX, CycloneDX, SWID — and defined practices and processes. Seven fields is a low bar, which is the point. It is low enough to be met and specific enough to be checked. Apply the same shape to a dataset release and the vague instruction to “record what went in” becomes an enumerable manifest, with the same questions asked of every input: who supplied it, which version, which identifier, what depends on it, who wrote the record, and when.

Encryption and access control reduce unauthorized disclosure. They do not establish lawful purpose, consent, minimization, or fairness. Privacy mechanisms can reduce exposure while leaving integrity attacks untouched. Governance assigns decision rights and review obligations; security implements controls against defined threats. The dataset program needs all three perspectives. Document residual risk and control coverage rather than claiming that one de-identification or encryption step makes the dataset safe.

The two anchors of this lesson sit on opposite sides of that line. Article 15(5) is a security obligation. It demands measures against data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks and model flaws. It says nothing about whether a given record should have been collected at all. The 3,226 suspected entries Stanford found in the LAION datasets were not an integrity failure. No attacker put them there, no hash was wrong, and a signed manifest over the bytes would have certified them faithfully. Curator-attached hashes and download-time verification would have preserved that content perfectly.

Choose a source that materially affects a production model. Identify who can create, alter, replay, label, or delete its records, and which trust boundaries each action crosses — including the ones nobody drew, such as which package index a resolver believes. Write two plausible attacks: one broad availability or integrity failure, one targeted manipulation. For each, identify preventive, detective, limiting, and recovery controls.

Then check your list against material that already exists. The six-agency sheet of 22 May 2025 names four mitigations: curator-attached raw-data hashes, hash verification at download with discard on mismatch, periodic re-scraping, and curator certification at publication. Do they apply to your source? If one does not, say why. Can your release manifest answer the seven NTIA minimum fields for each input? Would your monitoring have fired on 250 documents, or only on a percentage? Finish by naming the evidence that would allow investigators to bound affected predictions. If that evidence does not exist, the architecture is not incident-ready.