Skip to content
AI.info

Generative AI

Red-Teaming, Incident Readiness, and Release Governance

Organize risk-based red-teaming, release gates, incident playbooks, evidence preservation, and rollback for generative AI applications.

By the end you can

Key idea

Red-teaming is not a contest to make the model say something shocking

Serious campaigns ask a different question. Can a realistic adversary cause meaningful harm through the complete workflow? That workflow includes retrieval, tools, identities, permissions, rate limits, logs, interfaces, and downstream consumers.

The shocking-output theory has been measured at scale, and it lost. At DEF CON 31, in August 2023, 2,244 contestants attacked eight state-of-the-art LLMs in a Capture the Flag contest run by Humane Intelligence, Seed AI and AI Village. Each participant had 55 minutes. The 2024 transparency report counted the strategies people actually used.

The theatrical one returned nothing: “Across all the challenges, on the 580 queries asking the model to “ignore the previous instruction”, no one was successful.”

The mundane framings were the ones that worked. Of the 2,413 role-prompt queries beginning “You are a”, 9.8% succeeded. Of the 175 “step by step” chain-of-thought queries, 28% succeeded. Five hundred and eighty attempts at the famous attack, zero results. One hundred and seventy-five polite requests for reasoning, forty-nine of them through. The difference between a campaign that collects anecdotes and one that produces evidence is that the second one keeps a denominator.

Severity depends on how far the failure reaches, how reliably it repeats, how much harm it does, whether anyone notices it, and whether it can be contained. The goal is to reduce risk and to be ready to respond, not to collect dramatic outputs.

Test attack paths to assets and consequences, not isolated model embarrassment.

Visual

Red-teaming serves several stages of the lifecycle

The purpose and the evidence change as the system matures. Design review challenges assumptions, threat boundaries, intended use, and data flows before implementation. Discovery testing explores broad failure modes while architecture and controls are still flexible. Release testing attacks the candidate build against explicit gates and known incidents. Continuous testing repeats protected cases and generates new variants as dependencies change. Incident simulation rehearses detection, containment, communication, evidence, and recovery.

Run the same attack at two of these stages and you get two different artifacts. At discovery it is a design change. At release it is a gate decision with a date on it.

FigureTimeline · 5 stops

  1. Design review

    Challenge assumptions, threat boundaries, intended use, and data flows before implementation.

  2. Discovery testing

    Explore broad failure modes while architecture and controls remain flexible.

  3. Release testing

    Attack the candidate build against explicit gates and known incidents.

  4. Continuous testing

    Repeat protected cases and generate new variants as dependencies change.

  5. Incident simulation

    Rehearse detection, containment, communication, evidence, and recovery.

Comparison

Different teams contribute different evidence

Independence and domain expertise matter as much as adversarial creativity. Builders know the architecture, the assumptions, and the possible controls. They have fast access to internals and they own the remediation details. They also carry confirmation bias, which is the reason they should not be the sole approver. Independent testers challenge the system without sharing every design assumption. They expose blind spots and can test operational controls, and they need realistic access, a defined scope, and safe handling of findings. Domain and affected experts understand consequences, populations, norms, and practical misuse. They identify harms engineers miss and they sharpen severity assessment, and they need real authority in the release decision rather than a decorative review slot.

What "realistic access and scope" looks like when it is genuinely granted is on the public record. A pre-deployment version of OpenAI's o1 was tested by US AISI and UK AISI across biological, cyber and software/AI-development capabilities. The findings went to OpenAI ahead of the 5 December 2024 launch, and the report was published on 18 December 2024. On the 40-task Cybench benchmark, o1 solved 45% of tasks at Pass@10 against 35% for the best reference model, and 35% at Pass@1 against 30%.

The report also states the limit of its own access. The version tested “did not have the full set of mitigations that will be implemented in the publicly released version of the model”. Then it refuses, in writing, to be read as a clearance: “The results and conclusions herein should not be interpreted as an indication of whether any evaluated AI system or subcomponent thereof is safe or appropriate for release.”

That is the model for how independent evidence should be handed over. Findings, methods, and an explicit statement that the tester is not the approver.

Access itself gets arranged separately. On 29 August 2024 NIST announced memoranda of understanding with OpenAI and Anthropic establishing “the framework for the U.S. AI Safety Institute to receive access to major new models from each company prior to and following their public release”. Neither the o1 report nor that announcement says the o1 evaluation was conducted under the MoU. Treat them as two parallel facts about how access gets arranged, not as cause and effect.

FigureComparison · 3 columns

Builders

Know architecture, assumptions, and possible controls.

  • Fast access to internals
  • Risk of confirmation bias
  • Own remediation details
  • Should not be sole approver

Independent testers

Challenge the system without sharing every design assumption.

  • Expose blind spots
  • Need realistic access and scope
  • Can test operational controls
  • Require safe handling of findings

Domain and affected experts

Understand consequences, populations, norms, and practical misuse.

  • Identify harms engineers may miss
  • Improve severity assessment
  • Need authority in release decisions
  • Should not be treated as decorative reviewers

Example

A serious campaign contains multiple attack families

Select families based on the actual system and its users — and note that you are not inventing the list. Indirect prompt injection has its own entry, NISTAML.015, in NIST's adversarial machine learning taxonomy, published in March 2025 as NIST AI 100-2e2025. The taxonomy names the structural reason the attack works: “Because GenAI models combine the data and instruction channels, attackers can leverage the data channel to affect system operations by manipulating resources with which the system interacts.”

NIST records that such attacks “can result in violations across at least three categories of attacker goals: 1) availability violation, 2) integrity violation, and 3) privacy compromise”, and splits the chapter accordingly: availability attacks [NISTAML.016], integrity attacks [NISTAML.027], and privacy. OWASP's Top 10 for LLM Applications 2025 classifies the same class as LLM01:2025 Prompt Injection. Two standards bodies with numbered entries is a better starting point for a campaign plan than a whiteboard.

  • Instruction attacks: Direct, indirect, multilingual, encoded, visual, or multi-turn prompt injection. OWASP LLM01:2025 separates direct injection, where “a user's prompt input directly alters the behavior of the model in unintended or unexpected ways”, from indirect injection, which occurs when “an LLM accepts input from external sources, such as websites or files”.
  • Evidence attacks: Poisoned documents, stale sources, malicious metadata, citation laundering, and retrieval manipulation. This is the integrity half of NIST's split, integrity attacks [NISTAML.027], reached through the data channel rather than the prompt box.
  • Tool attacks: Unauthorized destinations, destructive arguments, transaction fraud, sandbox escape, and confused deputy behavior. This is what the combined data and instruction channels buy an attacker once the model can act.
  • Privacy attacks: Memorization probes, cross-tenant leakage, sensitive inference, and reviewer exposure. Privacy compromise is the third of NIST's three attacker goals.
  • Abuse attacks: Impersonation, fraud, harassment, malware assistance, and scaled deceptive content. The harm lands on people outside the system, and it rarely appears in the system's own logs.
  • Availability attacks: Long-context flooding, recursive tool loops, expensive generation, and queue starvation. NIST [NISTAML.016] names the concrete techniques. “Time-consuming background tasks” [NISTAML.017] is an injected prompt that makes the model loop before answering. “Inhibiting capabilities” tells the model it may not use certain APIs, for example the search API of an internet-connected chatbot. “Disruptive output formatting” covers homoglyph substitution in retrieved text, or forcing empty output with an <|endoftext|> token.

Steps

Convert a finding into an incident-ready control

A high-severity issue needs more than a prompt patch. The sequence runs in six steps. Confirm and scope: reproduce the path, the affected versions, users, assets, and preconditions. Classify severity by consequence, exploitability, scale, detectability, and reversibility. Contain exposure by disabling routes, revoking tools, narrowing permissions, or rolling back dependencies. Preserve evidence: traces, versions, inputs, outputs, reviewer actions, timestamps. Remediate in layers, across architecture, policy, validation, monitoring, and evaluation. Then validate and learn — retest variants, notify stakeholders, document decisions, add protected regressions.

The whole loop has been run once in public with the paperwork left visible. An indirect-injection path into Microsoft 365 Copilot was found by Aim Security, which named it EchoLeak. Microsoft published it on 11 June 2025 as CVE-2025-32711, titled “M365 Copilot Information Disclosure Vulnerability” and described as “Ai command injection in M365 Copilot allows an unauthorized attacker to disclose information over a network”, under CWE-74.

Step two, classification, is visibly a judgement rather than a measurement. Microsoft assigned a CVSS 3.1 base score of 9.3, Critical, with the vector AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N. NVD's own analysts scored the same record 7.5, High. One finding, two severity numbers, two competent bodies. That is why a release gate needs a severity procedure of its own rather than a field copied from a feed.

Step three had already happened by the time anyone outside could read about it. Microsoft records publiclyDisclosed=No, exploited=No and customerActionRequired=false, because the fix was applied service-side. The advisory says so plainly: “This vulnerability has already been fully mitigated by Microsoft. There is no action for users of this service to take. The purpose of this CVE is to provide further transparency.”

Note what that last sentence concedes. Nothing obliged the vendor to number a defect that customers could not act on. The number exists so that the failure mode becomes citable. That is the only form in which it can become somebody else's regression test.

FigureProcess · 6 steps

  1. 1. Confirm and scope

    Reproduce the path, affected versions, users, assets, and preconditions.

  2. 2. Classify severity

    Assess consequence, exploitability, scale, detectability, and reversibility.

  3. 3. Contain exposure

    Disable routes, revoke tools, narrow permissions, or roll back dependencies.

  4. 4. Preserve evidence

    Capture traces, versions, inputs, outputs, reviewer actions, and timestamps.

  5. 5. Remediate in layers

    Fix architecture, policy, validation, monitoring, and evaluation as required.

  6. 6. Validate and learn

    Retest variants, notify stakeholders, document decisions, and add protected regressions.

Key idea

A release gate must be able to say no

Governance is ineffective when every review approves the release with a few follow-up actions. The decision body needs explicit authority to limit traffic, remove capabilities, require human review, postpone launch, or reject the product.

The clearest demonstration that such authority is real comes from outside the builder. On 30 March 2023 the Italian data protection authority adopted provvedimento n. 112. It ordered, with immediate effect, a provisional limitation on OpenAI's processing of the personal data of people in Italy. ChatGPT went dark in Italy.

A recent incident sat among the circumstances the Garante's announcement listed: “ChatGPT, il più noto tra i software di intelligenza artificiale relazionale in grado di simulare ed elaborare le conversazioni umane, lo scorso 20 marzo aveva subito una perdita di dati (data breach) riguardanti le conversazioni degli utenti e le informazioni relative al pagamento degli abbonati al servizio a pagamento.” That is a 20 March 2023 breach touching users' conversations and paying subscribers' payment information.

The EDPB's ChatGPT Taskforce report of 23 May 2024 records that OpenAI implemented measures to comply with that urgent decision and with the subsequent decision of 11 April 2023. A live, already-deployed product was stopped on a dated order and restored on documented conditions. That is exactly the shape of a gate that can say no, applied after launch because nobody applied it before.

Record accepted residual risk, owners, monitoring, expiry dates, and conditions for reconsideration. Approval is a time-bounded decision about a specific configuration and use. It is not a permanent property of a model name.

A credible release process preserves the option not to deploy.

Case

Published dialogues, and a documented-testing duty already in force

A red-teaming methodology published in 2022 by Ganguli and colleagues at Anthropic tested 3 model sizes — 2.7B, 13B and 52B — across 4 model types. The abstract states what shipped with the method: “Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from.”

That sentence is checkable rather than rhetorical. The published red-team-attempts file in the Anthropic/hh-rlhf dataset on Hugging Face contains exactly 38,961 records, one per red-team attack. Each carries a transcript, a task description, a min-harmlessness score, a rating and strategy tags. Anyone can count them. A campaign nobody outside the team can inspect teaches nobody outside the team anything.

Obligation has since followed practice. Article 55(1) of Regulation (EU) 2024/1689, the EU AI Act, obliges providers of general-purpose AI models with systemic risk to “perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks;”. Point (c) of the same paragraph requires serious incidents to be reported to the AI Office without undue delay.

The word doing the work in (a) is documenting. An undocumented campaign does not discharge the duty, whatever it found.

The date is not a guess either. Article 113(b) makes Chapter V — the chapter that contains Article 55 — apply from 2 August 2025, ahead of the Regulation's general application date of 2 August 2026.

Readiness connects prevention with reversible operation

Generative systems will meet novel prompts, dependency changes, adversarial content, and unexpected user behavior. Red-teaming improves coverage. Incident readiness limits what the failures that get through can cost.

The four records in this lesson are the same lesson at four scales. A contest that measured which attacks worked instead of which looked good. A vendor that scored and contained a defect and then numbered it anyway. Two state institutes that published findings and disclaimed the authority to approve. A regulator that stopped a shipped product on a dated order. Every one of them left a document behind, and that is what made it usable by anyone else.

The capstone now asks for an end-to-end product design. It must integrate capability, evidence, tools, evaluation, security, serving, monitoring, and governance into one defensible release argument.

Key takeaways