AI literacy basics
Anatomy of an AI System
Trace the components of a deployed AI system from objectives and data to interfaces, policies, monitoring, and human accountability.
By the end you can
- Identify the major technical and human components of an AI system
- Explain why model performance does not determine system performance
- Recognize failures caused by interfaces, policies, incentives, and data flows
- Create a system inventory for an AI-enabled product
Analogy
The model is not the whole production
In a theater performance the lead actor is the visible part, and the production depends on the script, stage, lighting, rehearsal, ticketing, safety procedures, crew, and audience expectations.
A model is like the actor: important, sometimes remarkable, and unable to produce the experience alone. One difference matters. A stage performance happens once, while AI outputs can be copied and acted upon at enormous scale.
System quality depends on coordination among components, not only on the model’s talent.
Visual
From purpose to consequence
A deployed system connects design choices to real effects through a chain of links, and each link can improve, distort, or reverse the value of the model output.
- 1
Purpose and objectives
People define the problem, success criteria, constraints, and acceptable risk.
- 2
Data and knowledge
Records, labels, documents, sensors, rules, and feedback provide evidence.
- 3
Models and algorithms
Components transform inputs into scores, rankings, content, or action proposals.
- 4
Product and policy layer
Interfaces, thresholds, permissions, business rules, and fallbacks shape use.
- 5
Human workflow
Users interpret, approve, correct, override, or depend on the output.
- 6
Monitoring and governance
Teams measure behavior, investigate incidents, manage changes, and remain accountable.
A model answers; a system decides what the answer means
A model may output a 0.78 fraud risk score. The system decides whether to block the payment, request verification, queue a review, or do nothing.
That decision can depend on thresholds, customer history, jurisdiction, transaction value, reviewer capacity, and policy. Two organizations can deploy the same model and reach opposite outcomes, because their surrounding systems differ. Nothing about the model has changed.
The model transforms information; the system assigns consequence.
Case
The Epic Sepsis Model, validated by somebody else
The cost of that gap has been measured. The Epic Sepsis Model, a proprietary early-warning score that JAMA Internal Medicine described in 2021 as “implemented at hundreds of US hospitals”, was validated externally by Wong and colleagues at Michigan Medicine over 38,455 hospitalizations between 6 December 2018 and 20 October 2019. At an alerting threshold of a score of 6 or higher, the system raised an alert on 6,971 of those hospitalizations (18%) and still did not identify 1,709 of the patients who developed sepsis (67%); the model’s discrimination was an area under the ROC curve of 0.63 (95% CI, 0.62–0.64). Nothing about the model changes from hospital to hospital. The threshold, the alert routing, and the number of clinicians available to answer an alert do, and those are what a patient actually experiences.
Figure
Comparison
A strong prototype and a dependable service are different achievements
Laboratory evidence matters, but deployment adds users, latency, permissions, maintenance, changing data, and operational pressure.
Google engineers put a proportion on that difference. D. Sculley and nine colleagues published “Hidden Technical Debt in Machine Learning Systems” in 2015. Its central figure is captioned “Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex.” Later in the same paper they put a number on the surround. A mature system “might end up being (at most) 5% machine learning code and (at least) 95% glue code”. The prototype is usually the black box. The production system is everything drawn around it.
Prototype
Demonstrates a capability under selected conditions.
- Curated inputs and limited users
- Manual recovery may be acceptable
- Short evaluation horizon
- Goal: learn whether the idea is promising
Production system
Delivers a service repeatedly under real constraints.
- Messy inputs and diverse users
- Defined fallback and incident response
- Versioning, monitoring, and access control
- Goal: sustain value within a risk budget
Example
Three failures that no benchmark leaderboard can see
Even an accurate model can be embedded in a system that fails its users.
- Interface failure: a confidence score is displayed without explanation, so users treat a weak suggestion as a guarantee.
- Workflow failure: urgent cases are correctly flagged, but the review queue has no staffing to respond in time.
- Permission failure: a drafting assistant retrieves documents the user should not be allowed to access.
- Incentive failure: employees are rewarded for accepting more AI suggestions, discouraging necessary corrections.
- Monitoring failure: the model changes or the input population shifts, but no one notices the new error pattern.
Case
A recall filed against the interface, not the model
Interface failure has a recall number. On 12 December 2023 Tesla filed Part 573 Safety Recall Report 23V-838 with the US National Highway Traffic Safety Administration, covering 2,031,220 Model S, Model X, Model 3 and Model Y vehicles produced between 5 October 2012 and 7 December 2023. The defect described in that filing is not the driving model but the controls wrapped around it: “In certain circumstances when Autosteer is engaged, the prominence and scope of the feature’s controls may not be sufficient to prevent driver misuse of the SAE Level 2 advanced driver-assistance feature.” The remedy was correspondingly not a new model but “additional controls and alerts” delivered over the air — more prominent visual warnings, simpler engagement and disengagement, extra checks off controlled-access highways and when approaching traffic controls, and suspension from Autosteer for a driver who repeatedly fails to demonstrate continuous driving responsibility.
Key idea
AI systems are sociotechnical by construction
“Sociotechnical” means that technical behavior and social context shape each other: data reflects institutions, labels reflect judgments, interfaces influence trust, and organizational incentives determine how outputs are used.
This does not mean engineering is irrelevant. It means technical testing must include the people, procedures, and environments that determine impact.
The word is load-bearing in standards work, not only in seminars. The US National Institute of Standards and Technology released its Artificial Intelligence Risk Management Framework. That was on 26 January 2023. It states plainly: “AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior.” The framework defines seven trustworthiness characteristics. They are valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. NIST describes them there as “socio-technical system attributes”. None of them, that is to say, can be certified by looking at the model alone.
Context is not an external nuisance added after the model; it is part of the system being evaluated.
Steps
Build a system inventory before judging intelligence
A one-page inventory often reveals more than a long argument about the model’s apparent sophistication.
Lawmakers have written that inventory into binding text. Annex IV of the EU Artificial Intelligence Act sets out what a high-risk system’s technical documentation must contain “at a minimum”. It starts with the intended purpose, the provider and the version. It asks how the system interacts with hardware or software that is not part of the system itself. It asks for the design specifications, including “what the system is designed to optimise for”. It asks for the provenance and labelling procedures of the training data. It asks for an “assessment of the human oversight measures needed in accordance with Article 14”. It asks for the metrics used to measure accuracy and robustness, with “test logs and all test reports dated and signed by the responsible persons”. And it asks for the arrangements for monitoring the system in operation. The Regulation was published in the Official Journal of the European Union on 12 July 2024. It applies from 2 August 2026, with earlier and later dates for particular chapters.
- 1
List the actors
Identify developers, operators, reviewers, users, affected people, vendors, and accountable owners.
- 2
Trace the inputs
Record data sources, user prompts, retrieved knowledge, sensors, rules, and permissions.
- 3
Trace the outputs
Name scores, labels, content, alerts, and actions, including intermediate stages.
- 4
Map the controls
Document thresholds, validation, overrides, fallbacks, rate limits, and access checks.
- 5
Map the evidence
Locate evaluation results, logs, incident records, documentation, and unresolved assumptions.
- 6
Assign ownership
State who can change, pause, investigate, approve, and retire the system.
Key idea
Two cities published their inventory before anyone required it
Some cities published theirs before anyone required it. In September 2020 Amsterdam and Helsinki opened the first public municipal algorithm registers, each listing a deployed system alongside its datasets, its operating logic, its risks and whether a human reviews its output. “Algorithms play an increasingly important role in our lives,” said Touria Meliani, then Deputy Mayor of Amsterdam for the Digital City. “Together with the city of Helsinki, we are on a mission to create as much understanding about algorithms as possible and be transparent about the way we – as cities – use them.”
Three systems we will revisit
HarborHelp routes support messages, retrieves policy, and drafts replies for agents. FreshRoute predicts delivery times and recommends dispatch adjustments. LineSight detects possible defects in factory images and sends uncertain cases to inspectors.
These three recur through the course. They expose different interfaces between model output and human action, so reusing them lets the analysis deepen instead of restarting at a new toy example every time a concept appears.
Running examples create continuity: the same system looks different when examined through data, reliability, human factors, or risk.
Key takeaways
- A deployed AI system connects purpose, data, models, product controls, human workflows, monitoring, and governance.
- Models produce outputs, while surrounding policies and interfaces determine how those outputs affect people or processes.
- Prototype capability is not the same achievement as reliable production service.
- System failures can arise from permissions, queues, incentives, interfaces, or monitoring even when model metrics look strong.
- AI is sociotechnical because technical behavior and social context jointly shape outcomes.
- A system inventory should trace actors, inputs, outputs, controls, evidence, and ownership before deployment decisions are made.