Capstone: Audit an AI System End to End

HarborHelp is a proposed assistant for a subscription-software support team. It will classify incoming messages, retrieve internal policy, draft replies, and recommend refunds below a defined amount.

HarborHelp is invented; the proposal is not. On 27 February 2024 Klarna announced an assistant built with OpenAI. One month after going live globally, it “has had 2.3 million conversations, two-thirds of Klarna’s customer service chats”. It was “doing the equivalent work of 700 full-time agents”. It handled refunds, returns, cancellations and disputes in 23 markets and more than 35 languages. Errand resolution fell from 11 minutes to under 2. Klarna also reported a 25% drop in repeat inquiries. Fourteen months later the same company reopened recruitment for human agents. On 8 May 2025 its co-founder and chief executive, Sebastian Siemiatkowski, told Bloomberg that “as cost unfortunately seems to have been a too predominant evaluation factor when organizing this, what you end up having is lower quality.” Both the launch numbers and the correction are evidence. An audit that reads only the first is an audit of a press release.

Complete each page in order. Do not jump to model choice before the workflow and evidence are clear.

Page 4 is where audit workbooks most often record a control that does not exist. “Agents approve replies” is a design intention. The operational question is what approval looks like on the two-hundredth ticket of a shift. Medicine has measured the answer. Heleen van der Sijs, Jos Aarts, Arnold Vulto and Marc Berg surveyed the published studies of prescribing systems. Their review appeared in the Journal of the American Medical Informatics Association in March 2006. It is called “Overriding of Drug Safety Alerts in Computerized Physician Order Entry”. They found that “drug safety alerts are overridden by clinicians in 49% to 96% of cases”. Overriding is frequently justified, because many alerts do not apply to the patient in front of the prescriber. That is precisely why the number belongs on Page 4. A review step whose override rate nobody measures cannot be told apart from a review step that is not happening.

A strong decision memo considers alternatives rather than treating deployment as a yes-or-no vote on AI itself.

Three of the four options rest on the same load-bearing assumption. It is that a human reviewer catches what the system gets wrong. That assumption has been measured. Kate Goddard, Abdul Roudsari and Jeremy Wyatt showed 26 UK NHS general practitioners 20 prescribing scenarios with pre-validated answers. Six of the scenarios carried deliberately incorrect advice. Their paper appeared in the International Journal of Medical Informatics in 2014. It is called “Automation bias: empirical results assessing influencing factors”. Accuracy rose from 50.38% before the advice to 58.27% after it. At the same time, in 5.2% of all cases a decision that had been correct before the system spoke was wrong after it. The authors record “a net improvement of 8%”. Assistance helped on balance, and created a class of error the unassisted workflow did not have. A drafting-only pilot is the cheapest place to find out how large that class is for HarborHelp.

A capstone recommendation is strong when another reviewer can trace how evidence and values produced the conclusion.

The step from “Hypothesis” to “Pilot evidence” is not a formality. The base rate is published. Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu and Nils Pohlmann presented “Online controlled experiments at large scale” in 2013. All six were at Microsoft. They set out a tenet: “We are poor at assessing the value of ideas”. Behind it was a figure. “Only one third of the ideas tested at Microsoft improved the metric(s) they were designed to improve.” Kohavi put the same finding on the Bing engineering blog on 8 August 2013. “When ideas are evaluated objectively in a controlled experiment, less than a third move the metrics they were designed to improve.” Those were ideas teams believed in enough to build and ship into a test. A memo that lets the HarborHelp hypothesis stand in for the pilot result is betting against a base rate. One of the largest experimentation programmes in the industry measured that rate on itself.

A defensible recommendation names the proposed scope, baseline, expected value, evidence quality, unresolved risks, controls, owners, and next decision date. It also states what would change the recommendation.

Avoid “AI is ready” or “AI is too risky.” Use a bounded claim instead. For example: “Pilot retrieval and drafting for English and Italian billing tickets, with mandatory agent approval and predefined failure controls.”

Produce a concise artifact that someone outside the course could use in a real review meeting.

“Top ten failure modes” is not a course invention. It is failure modes and effects analysis, specified internationally in IEC 60812:2018. That edition, 3.0, was published on 10 August 2018. It covers hardware, software and processes, including human action. Its history also carries a warning about how you rank. For decades FMEA prioritised by Risk Priority Number: severity multiplied by occurrence multiplied by detection. That lets a trivial, frequent, easily detected failure outrank a rare and catastrophic one. Severity 3 with occurrence 10 and detection 4 scores 120. Severity 10 with occurrence 4 and detection 2 scores only 80. The joint AIAG & VDA FMEA Handbook, first edition issued June 2019, therefore “replaced the RPN (Risk Priority Number) with AP (Action Priority)”. AP is a lookup that grades every failure mode High, Medium or Low. It weights severity above occurrence, and occurrence above detection. Rank your ten failure modes; do not multiply them.

You can define AI operationally, distinguish major methods and task families, trace a system beyond its model, question data and objectives, design human handoffs, inspect reliability, and recognize risk.

Most importantly, you can make a bounded decision under uncertainty. That skill is the foundation for every later path, whether you study optimization, computer vision, MLOps, governance, or generative systems.