Skip to content
AI.info

Responsible AI

AI Security and Adversarial Machine Learning Governance

Build a threat-informed security program covering poisoning, evasion, extraction, privacy attacks, prompt injection, tool abuse, and supply-chain compromise.

By the end you can

Comparison

Ordinary software security, Adversarial ML testing, or Abuse-resistant system design?

Ordinary security protects the infrastructure. Adversarial testing probes the model. Abuse-resistant design limits what a successful attack accomplishes. The third is the one that still helps after detection fails.

FigureComparison · 3 columns

Ordinary software security

Protects code, identities, secrets, infrastructure, and data flows.

  • Essential foundation
  • Covers access control and secure development
  • May not model statistical or generative behavior
  • Cannot be replaced by AI testing

Adversarial ML testing

Examines model-specific manipulation and inference.

  • Tests poisoning, evasion, extraction, and leakage
  • Depends on realistic attacker assumptions
  • May miss workflow and tool abuse
  • Needs repeat testing after change

Abuse-resistant system design

Constrains consequences even when the model is manipulated.

  • Uses least privilege and independent authorization
  • Separates evidence from instruction
  • Supports containment and rollback
  • Reduces reliance on perfect detection

Example

AI security and adversarial risk under operational pressure

A customer-support agent retrieves internal documents. It can also issue refunds through tools. An attacker hides instructions inside a public webpage, and the retrieval system indexes that page. The agent reads the page as guidance rather than as evidence. It reveals policy fragments and proposes an unauthorized refund. No firewall was crossed. A document was read.

  • Trust-boundary failure: Retrieved content is treated as instruction rather than untrusted evidence.
  • Tool exposure: A model's proposal reaches a high-impact action path.
  • Data supply chain: Public content enters the corpus without adversarial screening.
  • Authorization gap: The tool layer trusts model intent instead of checking user rights and policy.
  • Detection challenge: The output reads fluently and passes ordinary quality evaluation.

The attack that arrives through content

An AI system can be attacked wherever something enters it. Data acquisition, training, the model artifact, the interface, the retrieval index. Then the tools, the infrastructure, the vendors, the users, and the outputs. The threats include poisoning, backdoors, evasion and prompt injection. They include model extraction, membership and attribute inference, data exfiltration and malicious tools. They include denial of service, insecure deserialization, compromised dependencies and insider abuse. A generic cybersecurity checklist misses the AI-specific paths. An AI-only review misses the fundamentals. Controls should start from assets, attacker capabilities, trust boundaries, abuse paths and what recovery would require.

Compromised dependencies are not hypothetical. Anyone who pip-installed PyTorch-nightly on Linux between 25 and 30 December 2022 got a malicious torchtriton package from PyPI instead of the real one. The advisory explains why: "Since the PyPI index takes precedence, this malicious package was being installed instead of the version from our official repository." The binary read /etc/hosts, /etc/passwd, $HOME/.gitconfig, $HOME/.ssh/* and the first 1,000 files in the user's home directory. It sent them out over encrypted DNS queries. The advisory went up on 31 December 2022, and MITRE ATLAS files the incident as AML.CS0015. No model was attacked here. The install path was.

Prompt injection is the other end of the same problem. It arrives through content rather than through a port. That is how a public webpage reached the support agent's refund tool.

Threat modeling that starts from ports will never reach the refund tool; start from the assets, the abuse paths, and what recovery would require.

Case

NIST AI 100-2 E2025, and the ATLAS matrix beside it

Two reference works now map this ground, and you can check a system against either. NIST finalised its adversarial machine learning taxonomy in March 2025 as AI 100-2 E2025. It separates predictive from generative systems, and maps attacker goals, capabilities and knowledge against lifecycle stage. MITRE's ATLAS covers the same territory as a public knowledge base of tactics and techniques actually used against AI systems, modelled on ATT&CK. One is a taxonomy. The other is a record of what attackers have already done.

Neither treats prompt injection as an exotic new category. OWASP ranks it first in its Top 10 for LLM applications, as LLM01:2025. Of retrieval-augmented generation and fine-tuning it says that "research shows that they do not fully mitigate prompt injection vulnerabilities". The 2026 edition, released on 4 August 2026, keeps prompt injection at number one. Microsoft said in July 2025 that indirect prompt injection is one of the most widely used techniques in the AI security vulnerabilities reported to it. It called this an inherent consequence of how the models work, not a bug awaiting a patch. Its answer is three layers: prevention, detection, and mitigation of impact.

It is content arriving where the system expected data. That is how a public webpage reached a refund tool.

Steps

Turn AI security and adversarial risk into an operating control

Assets and adversaries first. Then the boundaries. Then the authority the model is allowed to exercise. Detection comes last, because it is the layer most likely to fail.

Part of this is now law. Providers of high-risk systems must include, where appropriate, measures against data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks and model flaws. That is Article 15(5) of the EU AI Act. Look at the verbs it uses: preventing, detecting, responding to, resolving and controlling. Testing is not among them. Recital 76 says why the list is AI-specific, since attacks "can leverage AI specific assets, such as training data sets (e.g. data poisoning) or trained models (e.g. adversarial attacks or membership inference)". The Regulation was published in the Official Journal on 12 July 2024.

FigureProcess · 5 steps

  1. 1. Model assets and adversaries

    Define goals, capabilities, access, constraints, and likely abuse paths.

  2. 2. Draw trust boundaries

    Mark untrusted data, content, users, models, vendors, and tools.

  3. 3. Constrain authority

    Apply identity checks, least privilege, policy engines, isolation, and confirmations.

  4. 4. Test adaptive pathways

    Combine prompt, data, API, supply-chain, and operational attacks.

  5. 5. Detect, contain, recover

    Log traces, revoke credentials, quarantine data, roll back, notify, and patch.

Visual

Where AI security and adversarial risk enters the lifecycle

Five questions, in order. What does an attacker want? Where does trust change hands? Which surfaces can they reach? Which controls sit across those surfaces? And which tests keep the assumptions honest? Asked in that order, the controls follow from the threat rather than from a product catalogue.

FigureProcess · 5 steps
  1. 1

    Assets

    Sensitive data, model behavior, intellectual property, decisions, tools, credentials, and availability.

  2. 2

    Trust boundaries

    Users, retrieved content, plugins, vendors, training sources, and deployment services.

  3. 3

    Attack surfaces

    Data pipelines, prompts, APIs, model files, dependencies, and action channels.

  4. 4

    Layered controls

    Least privilege, isolation, validation, provenance, monitoring, and recovery.

  5. 5

    Assurance and learning

    Threat tests, red teams, incidents, patches, and updated assumptions.

Example

An attack tree from the outcome you fear

Pick one outcome you cannot afford. Trace it backward through the model, the tools, the users and the infrastructure.

  • Attack tree: Trace one high-impact outcome backward through model, tool, user, data, and infrastructure paths.
  • Untrusted-content drill: Inject adversarial text into retrieval sources and verify instruction-data separation.
  • Privilege audit: List every action the model can propose and every independent check before execution.
  • Recovery rehearsal: Rotate credentials, remove poisoned content, invalidate caches, and replay forensic logs.

Key idea

Passed the refusal benchmark, followed the webpage

A refusal benchmark measures the average prompt. An attacker picks the highest-consequence path instead, and keeps combining an ordinary weakness with an AI-specific one until something gives.

EchoLeak is what that looks like in production. Microsoft published CVE-2025-32711 on 11 June 2025. A single crafted email, carrying hidden instructions, was retrieved into Microsoft 365 Copilot's context and exfiltrated the user's data. The user clicked nothing. Microsoft rated it 9.3, critical; NVD's own analysis scored it 7.5. The exploit chained four bypasses. It evaded Microsoft's cross-prompt-injection classifier, and got around link redaction with reference-style Markdown. It used images the client fetched automatically, and abused a Teams proxy the content security policy allowed. Each of those controls worked. None of them survived being composed with the other three.

So plan for the attempt that succeeds. Minimize blast radius, preserve forensic evidence, make revocation fast, and refuse to grant the model authority it does not need. NIST's advice is to design on the assumption that the model will produce malicious output. The support agent passed its refusal benchmarks. Then it followed instructions that had arrived inside a public webpage, through the retrieval index rather than through a port.

Plan for the attempt that succeeds: blast radius, forensics and revocation speed are the controls that remain.

Threat assumptions expire with every change

An adaptive attacker is the assumption, so the threat model has a shelf life. Every change to the model, the tools or the suppliers expires it.

Name the attack path, the privilege finding or the recovery failure that would force the owner to redesign, restrict, remedy or retire the system. If nothing on the list could do that, the review was a description rather than a control.

Key takeaways