Skip to content
AI.info

Responsible AI

Red Teaming, Evaluation Governance, and Independent Challenge

Govern red teams, evaluation suites, independence, finding severity, remediation, retesting, and disclosure.

By the end you can

Example

Two days, public prompts, one reporting line

A model provider announces that its new system passed a red-team exercise. Read the exercise instead of the announcement. The team tested public prompts only. It had two days of access. It could not see the tool traces. And it reported what it found to the same leader who owned the launch date.

Nothing in that announcement is false. The exercise happened, the findings were real, the label is accurate. It simply could not have produced a finding that moved the launch.

  • Scope restriction: The tools, retrieval, memory and integrations that carry the real risk were out of bounds.
  • Independence conflict: The person who owns the launch date also decides what counts as severe and what gets published.
  • Coverage illusion: A short prompt list is reported as broad assurance.
  • Remediation gap: Findings are closed without reproducing the exploit or trying a variant of it.
  • Learning failure: A new attack never becomes a regression test or a design change.

Visual

Scope set by threat, not by convenience

Five things decide whether an exercise can find anything. What it is aimed at. Who is aiming it. What they can actually reach. What happens to what they find. And whether they can say it out loud the week before launch. Weaken one and the other four stop mattering.

FigureProcess · 5 steps
  1. 1

    Threat-led scope

    High-consequence assets, misuse cases, populations, and system integrations.

  2. 2

    Tester capability

    Domain, security, social, language, accessibility, and lived-experience expertise.

  3. 3

    Access and instrumentation

    Representative endpoints, tools, traces, policies, and environment.

  4. 4

    Finding lifecycle

    Evidence, severity, owner, remediation, retest, exception, and disclosure.

  5. 5

    Institutional independence

    Freedom to challenge, escalate, and preserve dissent against launch pressure.

Comparison

Benchmark evaluation, Internal red team, or Independent external challenge?

Benchmarks repeat, internal red teams go deep, and external challenge brings incentives the organization does not control. Only the last can report something the organization would rather not hear.

The third column has real examples. In November 2024 two national institutes tested a model together before its release for the first time. The US and UK AI Safety Institutes took Anthropic's upgraded Claude 3.5 Sonnet across four domains: biological capabilities, cyber capabilities, software and AI development, and safeguard effectiveness. The UK institute ran 47 cybersecurity challenges. Only 15 came off the shelf. The other 32 it had built privately, so the model could not have seen them. On those it reached a Pass@10 of 36% at 'cybersecurity apprentice' level, against 29% for the best reference model. The US institute used Cybench's 40 public challenges and measured 32.5%, against 35%. Both found the tested safeguards could be routinely circumvented. Neither certified anything.

The first column comes with a warning. Scale AI commissioned 1,205 fresh grade-school maths problems, matched to GSM8k on how often humans solve them, how many steps they take, and how big the answers are. Accuracy fell by up to 8 percentage points on the new set. The more likely a model was to generate a GSM8k example, the wider its gap — a Spearman r-squared of 0.36. Part of what a leaderboard measures is memory of the leaderboard.

FigureComparison · 3 columns

Benchmark evaluation

Measures defined tasks on controlled cases.

  • Repeatable and comparable
  • Limited to known distributions
  • Can be optimized or contaminated
  • Good for regression, not exhaustive assurance

Internal red team

Explores adaptive failures with system context.

  • Fast access to builders
  • Can iterate deeply
  • May share organizational incentives
  • Needs protected escalation

Independent external challenge

Brings separate incentives and perspectives.

  • Can expose blind spots and legitimacy gaps
  • Requires safe access and disclosure rules
  • May lack full system context
  • Should complement, not replace, internal testing

Steps

From the feared outcome to the regression test

Start at the end. Name the outcome you are afraid of, then work backward: who would have to reach what, which test would surface it, who decides how bad it is, and what regression keeps it from coming back.

Securing AI systems is never finished. That is the eighth lesson Microsoft's AI Red Team drew from red-teaming more than 100 generative AI products since 2021. The loop has no exit.

FigureProcess · 5 steps

  1. 1. Define assurance questions

    Identify feared outcomes, attacker models, affected groups, and release claims.

  2. 2. Establish independent access

    Provide representative systems, traces, tools, and protected escalation.

  3. 3. Execute adaptive tests

    Combine known cases, creative exploration, social context, and system composition.

  4. 4. Govern findings

    Record reproducible evidence, severity, owner, deadline, exception, and disclosure.

  5. 5. Retest and institutionalize

    Verify fixes, test variants, add regressions, and update design and policy.

Analogy

A fire drill designed by the building owner and observed by firefighters

When the building owner picks which exits may be tested and then grades the result, the building passes. Independent firefighters walk the same corridors and find the locked door.

The analogy breaks in one place, and the break is the point. A building has a fixed number of exits. An AI system's exits are opened by language, by retrieved content, by user behavior, and by the next model update. A drill run once against a fixed script tests the script.

Independent challenge matters when it can test the real system and change the release decision.

Regression coverage is what proves it closed

Red teaming is structured adversarial inquiry: you go looking for the failure, the abuse path, the control that does not hold. The label is easy to obtain. What makes it worth anything is threat relevance, real access, testers who are not all the same person, independence, captured evidence, the authority to force a fix, and a retest. The exercise at the top of this lesson had the label. It had two days and a reporting line back to the person who owned the launch date.

Which attack you try matters more than how hard you try. Of the 580 attempts that told the model to 'ignore the previous instruction', none succeeded. Of the 175 that walked it through 'step by step', 28% did. Those attempts were made at DEF CON 31 in 2023, where 2,244 contestants got 55 minutes each in front of eight state-of-the-art large language models. It was run by Humane Intelligence, SeedAI and AI Village, with technical input from the White House Office of Science and Technology Policy. The crude attack is the one everybody has already defended against.

Evaluation governance is the boring half. Development diagnostics, release gates, independent challenge, external research and post-deployment monitoring are five different jobs. Run them out of one test suite and none of them is being done. Test sets need versions and contamination control. Findings need severity criteria, owners, deadlines, and a named person who accepted the risk when nobody fixed it. A published starting list beats an improvised one: NIST's Generative AI Profile set out more than 200 suggested actions across twelve generative-AI risk categories in July 2024. None of that is benchmarking, and red teaming is not benchmarking either. That is the third of the eight lessons Microsoft's AI Red Team reported in January 2025. The two are not substitutes.

Regression coverage is the item teams drop first. Drop it and a closed exploit returns a quarter later as a fresh finding, and nobody notices it is the same one. Findings can outlive the exercise that produced them. In 2022 Anthropic released the transcripts of 38,961 red team attacks as a public dataset. The work tested models at 2.7B, 13B and 52B parameters, and reported that models trained with RLHF got harder to red team as they got bigger. A corpus you can rerun is a different asset from a report you can file.

A body outside the developer is the structural point, not the testing technique. The UK set one up in November 2023, announced at the Bletchley Park summit. In February 2025 it renamed the AI Safety Institute the AI Security Institute, as the remit narrowed towards security risks. Europe has since written the practice into law. Article 55(1)(a) of the AI Act, published in the Official Journal in July 2024, requires providers of general-purpose AI models with systemic risk to run adversarial testing and to document it. Independent challenge still only counts when the challenger can publish a finding the vendor dislikes.

Regression coverage is what proves the finding was actually closed.

Test sets that leak into training, and findings logged without owners or deadlines, quietly turn a release gate into a development diagnostic.

Key idea

Where a red teaming and independent evaluation control can still fail

Red teaming becomes theater in three ways. The scope excludes the functions that matter. The findings go into a confidential report and stop there. Or the exercise runs after the approval it was supposed to inform. A fourth is quieter: a pile of harmless findings buries one systemic control failure, because volume reads as diligence.

The organisers of the DEF CON challenge were honest about their own ceiling. They were not permitted to compare how the models behaved before and after safety interventions. So they say plainly what follows: the exercise cannot validate that a model was successfully fixed, or that it will not cause harm in deployment. That report also states that it was released without company legal or communications sign-off. Both admissions are in it because someone was free to put them there.

Often nobody is free to. The terms of service and enforcement strategies of major AI developers deter good-faith safety evaluation. Researchers fear account suspension or legal reprisal for doing the work, or for publishing it. Twenty-three researchers led by Shayne Longpre at MIT documented that in 2024. Company-run access programmes are not a substitute: they have limited community representation, receive inadequate funding, and are not independent of the corporate incentives they exist to check.

No red team enumerates every attack, culture, language, context or future adaptation. Treat findings as evidence about mechanisms and process quality, never as a checklist of prompts now solved. Two days of access, public prompts only, a report to the person who owned the launch date: that exercise could only ever find what it had been allowed to find.

Read the scope, the timing and the reporting line before the findings; what an exercise was barred from testing says more than what it found.

A finding is worth its remediation record

Findings are the output, and a finding is worth exactly what the remediation record says it is. Nothing more.

The machinery for fixing that exists on paper. A 2025 follow-on to the safe-harbour work proposed standardised AI flaw reports with rules of engagement. It proposed broadly-scoped disclosure programmes modelled on bug bounties and backed by legal safe harbours. And it proposed shared infrastructure to route a report to everyone it affects. The urgency is transfer: jailbreaks and other flaws found in one provider's general-purpose system often work on another's.

So set the threshold now. Name the severity, the retest result, or the escalation that would force this system to be redesigned, restricted, remedied or retired. If no outcome would, you are not evaluating. You are collecting reassurance.

Key takeaways