AI literacy basics
Anatomy of an AI System
Trace the components of a deployed AI system from objectives and data to interfaces, policies, monitoring, and human accountability.
By the end you can
- Identify the major technical and human components of an AI system
- Explain why model performance does not determine system performance
- Recognize failures caused by interfaces, policies, incentives, and data flows
- Create a system inventory for an AI-enabled product
Analogy
The model is not the whole production
In a theater performance the lead actor is the visible part, and the production depends on the script, stage, lighting, rehearsal, ticketing, safety procedures, crew, and audience expectations.
A model is like the actor: important, sometimes remarkable, and unable to produce the experience alone. One difference matters. A stage performance happens once, while AI outputs can be copied and acted upon at enormous scale.
System quality depends on coordination among components, not only on the model’s talent.
Visual
From purpose to consequence
A deployed system connects design choices to real effects through a chain of links, and each link can improve, distort, or reverse the value of the model output.
- 1
Purpose and objectives
People define the problem, success criteria, constraints, and acceptable risk.
- 2
Data and knowledge
Records, labels, documents, sensors, rules, and feedback provide evidence.
- 3
Models and algorithms
Components transform inputs into scores, rankings, content, or action proposals.
- 4
Product and policy layer
Interfaces, thresholds, permissions, business rules, and fallbacks shape use.
- 5
Human workflow
Users interpret, approve, correct, override, or depend on the output.
- 6
Monitoring and governance
Teams measure behavior, investigate incidents, manage changes, and remain accountable.
A model answers; a system decides what the answer means
A model may output a 0.78 fraud risk score. The system decides whether to block the payment, request verification, queue a review, or do nothing.
That decision can depend on thresholds, customer history, jurisdiction, transaction value, reviewer capacity, and policy. Two organizations can deploy the same model and reach opposite outcomes, because their surrounding systems differ. Nothing about the model has changed.
The model transforms information; the system assigns consequence.
Case
The Epic Sepsis Model, validated by somebody else
The cost of that gap has been measured. The Epic Sepsis Model, a proprietary early-warning score that JAMA Internal Medicine described in 2021 as “implemented at hundreds of US hospitals”, was validated externally by Wong and colleagues at Michigan Medicine over 38,455 hospitalizations between 6 December 2018 and 20 October 2019. At an alerting threshold of a score of 6 or higher, the system raised an alert on 6,971 of those hospitalizations (18%) and still did not identify 1,709 of the patients who developed sepsis (67%); the model’s discrimination was an area under the ROC curve of 0.63 (95% CI, 0.62–0.64). That last figure describes the model itself. It is the chance that a randomly chosen patient who developed sepsis scored higher than a randomly chosen one who did not: 0.5 is a coin toss and 1.0 is perfect. The bracket, the 95% confidence interval, is the range the true value plausibly lies in, given the size of the study. Epic had reported 0.76 to 0.83. So the first finding is about the model: its claimed performance did not hold up when someone else tested it, at one academic health system. The second is about the system around it. The 18% and the 67% came from one choice of threshold, a score of 6. A lower cut-off raises more alerts and misses fewer patients; a higher one does the reverse. Another health system that tested the model on its own patients chose 5. The threshold, the alert routing, and the number of clinicians available to answer an alert are set hospital by hospital, and those are what a patient actually experiences.
FigureInteractive
Two proportion bars over the Michigan Medicine validation of the Epic Sepsis Model: alerts raised on 18% of 38,455 hospitalizations, and 67% of the patients who developed sepsis not identified. Below, discrimination of 0.63 with a 95% confidence interval of 0.62 to 0.64 marked on a scale from 0.50 to 1.00.
Comparison
A strong prototype and a dependable service are different achievements
Laboratory evidence matters, but deployment adds users, latency, permissions, maintenance, changing data, and operational pressure.
Google engineers put a proportion on that difference. D. Sculley and nine colleagues published “Hidden Technical Debt in Machine Learning Systems” in 2015. Its central figure is captioned “Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex.” Later in the same paper they put a number on the surround. A mature system “might end up being (at most) 5% machine learning code and (at least) 95% glue code”. The prototype is usually the black box. The production system is everything drawn around it.
Prototype
Demonstrates a capability under selected conditions.
- Curated inputs and limited users
- Manual recovery may be acceptable
- Short evaluation horizon
- Goal: learn whether the idea is promising
Production system
Delivers a service repeatedly under real constraints.
- Messy inputs and diverse users
- Defined fallback and incident response
- Versioning, monitoring, and access control
- Goal: sustain value within a risk budget
Example
Five failures that no benchmark leaderboard can see
Even an accurate model can be embedded in a system that fails its users.
- Interface failure: a confidence score is displayed without explanation, so users treat a weak suggestion as a guarantee.
- Workflow failure: urgent cases are correctly flagged, but the review queue has no staffing to respond in time.
- Permission failure: a drafting assistant retrieves documents the user should not be allowed to access.
- Incentive failure: employees are rewarded for accepting more AI suggestions, discouraging necessary corrections.
- Monitoring failure: the model changes or the input population shifts, but no one notices the new error pattern.
Case
A recall filed against the interface, not the model
Interface failure has a recall number. On 12 December 2023 Tesla filed Part 573 Safety Recall Report 23V-838 with the US National Highway Traffic Safety Administration, covering 2,031,220 Model S, Model X, Model 3 and Model Y vehicles produced between 5 October 2012 and 7 December 2023. The defect described in that filing is not the driving model but the controls wrapped around it: “In certain circumstances when Autosteer is engaged, the prominence and scope of the feature’s controls may not be sufficient to prevent driver misuse of the SAE Level 2 advanced driver-assistance feature.” The remedy was correspondingly not a new model but “additional controls and alerts” delivered over the air — more prominent visual warnings, simpler engagement and disengagement, extra checks off controlled-access highways and when approaching traffic controls, and suspension from Autosteer for a driver who repeatedly fails to demonstrate continuous driving responsibility.
Key idea
AI systems are sociotechnical by construction
“Sociotechnical” means that technical behavior and social context shape each other: data reflects institutions, labels reflect judgments, interfaces influence trust, and organizational incentives determine how outputs are used.
This does not mean engineering is irrelevant. It means technical testing must include the people, procedures, and environments that determine impact.
The word is load-bearing in standards work, not only in seminars. The US National Institute of Standards and Technology released its Artificial Intelligence Risk Management Framework. That was on 26 January 2023. It states plainly: “AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior.” The framework defines seven trustworthiness characteristics. They are valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. NIST describes them there as “socio-technical system attributes”. None of them, that is to say, can be certified by looking at the model alone.
Context is not an external nuisance added after the model; it is part of the system being evaluated.
Steps
Build a system inventory before judging intelligence
A one-page inventory often reveals more than a long argument about the model’s apparent sophistication.
- 1
List the actors
Identify developers, operators, reviewers, users, affected people, vendors, and accountable owners.
- 2
Trace the inputs
Record data sources, user prompts, retrieved knowledge, sensors, rules, and permissions.
- 3
Trace the outputs
Name scores, labels, content, alerts, and actions, including intermediate stages.
- 4
Map the controls
Document thresholds, validation, overrides, fallbacks, rate limits, and access checks.
- 5
Map the evidence
Locate evaluation results, logs, incident records, documentation, and unresolved assumptions.
- 6
Assign ownership
State who can change, pause, investigate, approve, and retire the system.
Example
LineSight’s inventory on one page
LineSight is one of the three invented systems this lesson closes on. It photographs parts on a factory line, flags possible defects, and sends the uncertain ones to human inspectors. Here is its inventory, step by step. Every detail is invented; the shape is what carries over.
- Actors. The plant’s data team, which trained the model and runs it. The line inspectors, who clear the uncertain pile. The quality manager, who answers for the defect rate. And the customers who receive every part that passed.
- Inputs. Photographs from two cameras at each station, the part number from the line’s database, and the lighting settings. The inspectors’ corrections are stored, not fed back in; they wait for the next retraining.
- Outputs. A defect score for each photograph, then one of three actions: pass, scrap, or send to an inspector. The score is intermediate. The action is what moves a part.
- Controls. Two thresholds split the three actions. A part whose photograph cannot be read goes to an inspector, never to pass. The uncertain pile is capped at what two inspectors can clear in a shift; above that, the line slows down.
- Evidence. The held-out test from training, a two-week trial on this line, and the inspectors’ correction log. Missing: any test after the camera replacement planned for next year.
- Ownership. The data team may retrain the model and move the thresholds. Only the quality manager may switch the system off. Nobody has been named to approve a new version before it replaces the old one.
Half a page, and it has already found a missing test and a missing owner, before anyone argued about the model.
The inventory European law requires
Lawmakers have written that inventory into binding text. Article 11(1) of the EU Artificial Intelligence Act requires a high-risk system’s technical documentation to contain, “at a minimum”, the elements set out in Annex IV. The Annex lists what that documentation “shall contain at least”. It starts with the intended purpose, the provider and the version. It asks how the system interacts with hardware or software that is not part of the system itself. It asks for the design specifications, including “what the system is designed to optimise for”. It asks for the provenance and labelling procedures of the training data. It asks for an “assessment of the human oversight measures needed in accordance with Article 14”. It asks for the metrics used to measure accuracy and robustness, with “test logs and all test reports dated and signed by the responsible persons”. And it asks for the arrangements for monitoring the system in operation. The Regulation was published in the Official Journal of the European Union on 12 July 2024, and its general date of application was 2 August 2026, but not for this part: Article 11 and Annex IV are among the high-risk requirements held back by the postponement set out in lesson 1.
Key idea
Two cities published their inventory before anyone required it
Some cities published theirs before anyone required it. In September 2020 Amsterdam and Helsinki opened the first public municipal algorithm registers, each listing a deployed system alongside its datasets, its operating logic, its risks and whether a human reviews its output. “Algorithms play an increasingly important role in our lives,” said Touria Meliani, then Deputy Mayor of Amsterdam for the Digital City. “Together with the city of Helsinki, we are on a mission to create as much understanding about algorithms as possible and be transparent about the way we – as cities – use them.”
Three systems to keep in view
HarborHelp routes support messages, retrieves policy, and drafts replies for agents. FreshRoute predicts delivery times and recommends dispatch adjustments. LineSight detects possible defects in factory images and sends uncertain cases to inspectors.
HarborHelp returns in the capstone, where you audit it end to end; FreshRoute and LineSight are yours to reuse as you go. The three expose different interfaces between model output and human action, so reusing them lets the analysis deepen instead of restarting at a new toy example every time a concept appears.
Running examples create continuity: the same system looks different when examined through data, reliability, human factors, or risk.
Key takeaways
- A deployed AI system connects purpose, data, models, product controls, human workflows, monitoring, and governance.
- Models produce outputs, while surrounding policies and interfaces determine how those outputs affect people or processes.
- Prototype capability is not the same achievement as reliable production service.
- System failures can arise from permissions, queues, incentives, interfaces, or monitoring even when model metrics look strong.
- AI is sociotechnical because technical behavior and social context jointly shape outcomes.
- A system inventory should trace actors, inputs, outputs, controls, evidence, and ownership before deployment decisions are made.