AI Agents Capstone: Design, Evaluate, and Govern an Agentic System

Task and authority, Runtime architecture, Control surface, Evaluation program, and Lifecycle governance are the five answers a capstone gives. In that order. The Evaluation program and Lifecycle governance should have separate owners and separate tests.

Each of the five can be answered with a document a reviewer can fetch, not with a slide. A federal rule that fixes the decision deadlines. A benchmark that reports how often the same agent repeats itself on the same task. A protocol specification that states in its own text what it cannot enforce. A security advisory with a score attached. A regulation that names who must be able to switch the system off. The sections below use those documents instead of describing them.

The system must read clinical and insurance documents, identify missing evidence, communicate with staff, draft submissions, monitor payer responses, and update case state. Errors can delay care, expose health data, or submit unauthorized claims.

The clock is not a design choice. A payer has 72 hours to decide an expedited request and seven calendar days to decide a standard one, extendable to 14 days in certain circumstances. Those windows were fixed by the CMS Interoperability and Prior Authorization Final Rule (CMS-0057-F), issued 17 January 2024. The CMS fact sheet puts the deadline in one sentence: “We are requiring impacted payers (excluding QHP issuers on the FFEs) to send prior authorization decisions within 72 hours for expedited (i.e., urgent) requests and seven calendar days for standard (i.e., non-urgent) requests.”

The same rule adds three duties to the clock. From 2026 every denial needs a specific reason. Prior authorization metrics become an annual public report, the first due by 31 March 2026. The Prior Authorization API is required from 1 January 2027. A capstone in this domain inherits a deadline, a duty to explain each refusal, an audit trail and a public number. All of that before anyone draws a box.

The capstone treats an agent as a sociotechnical control system. The design must prove that the chosen autonomy creates value beyond a simpler workflow, while staying observable, bounded, reversible, and contestable.

The best conclusion may be a deterministic workflow, a model-assisted process, a staged agent, or no deployment. The evidence determines the architecture. Not the desire to showcase autonomy.

That comparison has been run under control, and it came out backwards. Sixteen experienced open-source developers worked 246 real issues in their own mature repositories, each issue randomized to allow or to disallow early-2025 AI tools. Before starting, the developers forecast a 24% speed-up. After finishing the work, they still believed they had been sped up by 20%. The measurements said otherwise. METR published the trial on 10 July 2025, and its Core Result section reads: “When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts.” Outside forecasters were further out than the participants: economics experts had predicted 39% shorter, ML experts 38% shorter.

Nobody in that trial was lying, and no amount of asking would have found the slowdown. That is the whole argument for running the comparison rather than collecting impressions of it.

A Model-assisted workflow, a Staged case agent, and an Autonomous case agent are three defensible answers, and the first is the baseline the others must beat. The choice is defensible when the agent beats that baseline on a representative and repeated task suite, measured on the same cases more than once. It is not defensible when the case for the agent rests on how the architecture looks on a slide.

“More than once” is not a rhetorical flourish. It has a published price. τ-bench, released by Sierra in 2024, introduced the pass^k metric: the chance that all k i.i.d. trials of the same task succeed, averaged across tasks. Its abstract reports what that costs: “Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail).” The single-run averages were 61.2% in retail and 35.2% in airline, under 50% across the two domains. In retail, the same gpt-4o agent that clears 60% on one run drops below 25% once all eight runs have to succeed. Demanding eight consistent runs cuts the score by more than half.

A comparison table built from single runs is therefore not a close approximation of the repeated one. It is a different quantity. And it flatters the more autonomous option, because that is the option with the most places to vary.

The evidence pack is produced for one real system rather than for a template. Build it so that a missing result is visible. A drawing of every component proves nothing on its own, and the pack should make it obvious wherever release evidence is absent. What the pack stands or falls on is the comparison against the simpler option, run on cases that represent the real work and run more than once.

The threat-modelling step is a run rather than a review, because an executable version of it already exists. AgentDojo is an executable environment for prompt-injection testing of tool-using agents, presented at NeurIPS in 2024. Its abstract lists what is inside: “We populate the environment with 97 realistic tasks (e.g., managing an email client, navigating an e-banking website, or making travel bookings), 629 security test cases, and various attack and defense paradigms from the literature.” Two of its results have to be carried separately. State-of-the-art LLMs fail many of those tasks even with no attack present. And existing injection attacks break some security properties but not all.

Neither result is obtainable from a diagram. The first says the agent is already unreliable before anyone attacks it. The second says a passing defence has been tested against the attacks that existed when the suite was written, which is a claim with a date on it.

The final recommendation must name the architecture, evidence, residual risk, owner, release stage, and the conditions that trigger pause, redesign, or non-deployment. For a system of this kind, some of those conditions are already written down. AI systems used for risk assessment and pricing in life and health insurance are high-risk under the Artificial Intelligence Act, Regulation (EU) 2024/1689, at Annex III, point 5(c).

Article 14(4) then requires that human overseers be enabled “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;”. They must also be able to decide not to use the system, to disregard, override or reverse its output, and to interrupt it through a 'stop' button. A deployer with reason to think the system presents a risk must tell the provider and the market surveillance authority without undue delay, and suspend the use of that system. That is Article 26(5). Article 26(6) requires automatically generated logs to be kept for at least six months.

Override, refusal, interruption, suspension, retention. The statute names the same capabilities this section asks a capstone to name. A design without them is not merely thin.

A reviewer starts with the warning rather than with the diagram: “A comprehensive architecture diagram can hide the absence of release evidence”. So before studying the boxes and arrows, they ask what the system was measured against and what the measurements showed. Whether it did better than the simpler baseline, on cases that look like the real ones, across more than a single run.

The capstone is complete only when another team can inspect the design, reproduce the evaluation, and challenge the release decision. Every deliverable below is written so that its absence is visible. Each one names a figure, a duty, or a suite that either exists in the pack or does not.

A responsible capstone defines evidence thresholds before choosing whether to deploy the agent more broadly, and states them as numbers rather than adjectives. Reliability is declared at pass^k on the same cases. A threshold set on single runs would have passed the retail agent at over 60% while its pass^8 sat below 25%. Rollback and ownership are drilled or they are unproven. There is no third state.

The gate that matters most is the one the committee in “Refusal of Recovery” did not have: a rule fixed in advance that a rise in adverse determinations blocks the release however much handle time improves. Written afterwards, that rule is an opinion. Written before the test, it is a stop condition.

Uncertainty and rejected alternatives belong in the final presentation, rather than a case for the most complex architecture.

Two of the five steps are where presentations usually fail. Presenting the evidence means showing variance rather than the best trajectory. The distance between a single run and pass^8 is what the board is entitled to see, and in τ-bench's retail domain that distance ran from over 60% to below 25%. Naming residual risk means saying which harms were never measured, and how the team knows what it thinks it knows. The METR participants believed they had been sped up 20% while the measured time had risen 19%. A pilot team's self-report is not evidence of value, and should be presented as what it is.