AI literacy basics
Reliability Under Uncertainty
Understand probabilistic outputs, confidence, calibration, edge cases, distribution shift, and the design of graceful failure.
By the end you can
- Explain why AI reliability must be measured across conditions and error types
- Distinguish a confidence-like score from a guarantee of correctness
- Recognize edge cases, out-of-scope inputs, and distribution shift
- Design basic failure handling through detection, fallback, containment, and recovery
Analogy
A forecast is evidence, not a promise
A weather forecast showing a seventy percent chance of rain does not promise rain at your exact location, and one dry day does not prove the forecast method is useless.
AI scores often have a similar probabilistic role: they summarize evidence under a model and a data distribution. A published forecast is a calibrated probability. Many model scores are not, and can be hard to interpret across populations.
A number between zero and one is not automatically a trustworthy probability.
Example
Reliability is a pattern of errors, not one average
Two systems with the same overall accuracy can behave very differently in practice.
The speech-recognition case below has been measured. Allison Koenecke and colleagues published a study in the Proceedings of the National Academy of Sciences on 23 March 2020. They put matched recorded interviews through five commercial recognizers: Amazon, Apple, Google, IBM and Microsoft. The speakers were matched on age and gender and said the same words. Averaged across the five systems, the word error rate was 0.35 for black speakers and 0.19 for white speakers. For black men it was 0.41. Stanford’s account of the work describes the tail of that distribution. Transcripts wrong enough to be unusable “occurred in over 20 percent of samples spoken by blacks, versus fewer than 2 percent of samples spoken by whites”. One headline accuracy figure for any of those five systems would have concealed every part of that.
- A defect detector may miss tiny cracks while performing well on large surface damage.
- A speech recognizer may degrade with background noise, uncommon names, or unfamiliar accents.
- A fraud model may create more false alarms for new customers with limited history.
- A language model may answer common questions well but invent details when sources are absent.
- A routing system may work during normal demand and fail during storms, strikes, or major events.
Figure
Reliability requires knowing where errors cluster, who experiences them, and what happens next.
Comparison
Score, confidence, and calibration are not interchangeable
Interfaces often display a number without explaining what it means. That ambiguity can create false certainty.
The gap between the three is measurable. It widened as models got more accurate. Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger published “On Calibration of Modern Neural Networks”. That was in 2017. Their finding: “We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated.” Their Figure 1 sets a 5-layer LeNet from 1998 against a 110-layer ResNet from 2016, on the CIFAR-100 image set. The ResNet is much the better classifier, at 30.6% error against 44.9%. It is much the worse probability estimator. “The average confidence of LeNet closely matches its accuracy, while the average confidence of the ResNet is substantially higher than its accuracy.” The paper also shows the two can be separated after training. It finds that “temperature scaling – a single-parameter variant of Platt Scaling – is surprisingly effective at calibrating predictions”. Accuracy and calibration are two different things to buy. Buying one does not deliver the other.
Model score
A numerical output used to rank, classify, or trigger a policy.
- May be a logit, similarity, risk, or probability-like value
- Meaning depends on training and transformation
- Useful for ordering or thresholding
- Not inherently calibrated
Confidence claim
A statement about how certain the system appears to be for a case.
- May derive from score separation or model agreement
- Can be wrong on unfamiliar inputs
- Needs a clear interpretation
- Should not be anthropomorphized
Calibration
A relationship between predicted probabilities and observed frequencies over many comparable cases.
- Evaluated across groups and conditions
- Can drift after deployment
- Does not guarantee individual correctness
- Useful for risk-based decisions
Visual
Reliability includes what happens after uncertainty appears
No system detects every possible error. A resilient one does not try. It builds layers that reduce exposure and make recovery possible.
- 1
Detect abnormal conditions
Check missing inputs, unfamiliar formats, weak evidence, conflicting signals, and out-of-range values.
- 2
Limit the action
Choose a reversible output, smaller scope, lower authority, or no action when confidence is insufficient.
- 3
Route or fall back
Send the case to a person, simpler rule, alternative service, or request for more information.
- 4
Record the event
Capture input context, model version, decision path, and user response for diagnosis.
- 5
Recover and learn
Correct the immediate outcome, then decide whether model, policy, data, or interface changes are justified.
Distribution shift: the world stops resembling the evidence
Models are evaluated on samples drawn from particular populations, periods, devices, and workflows, and deployment can differ because user behavior changes, new products appear, sensors are replaced, policies shift, or incentives adapt.
Distribution shift does not guarantee failure. Stable averages do not guarantee safety. Teams have to monitor the variables and slices that matter for the decision, not merely whether overall volume changed.
Generalization has a boundary; deployment tests whether the current world remains inside it.
Case
Google Flu Trends, and the world moving underneath it
Google Flu Trends is the case that made this concrete. It estimated influenza-like illness from search queries, and for a while it tracked the US Centers for Disease Control and Prevention closely. Then it did not. Writing in Science on 14 March 2014, David Lazer, Ryan Kennedy, Gary King and Alessandro Vespignani reported that “GFT overestimated the prevalence of flu in the 2012–2013 season and overshot the actual level in 2011–2012 by more than 50%. From 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence 100 out of 108 weeks.” Their diagnosis names a mechanism this section describes. They call it “algorithm dynamics”: “the changes made by engineers to improve the commercial service and by consumers in using that service”. The search engine being measured was itself under continuous revision — “the official Google search blog reported 86 changes in June and July 2012 alone”. Nothing inside the model broke; the instrument it was reading moved. On 20 August 2015 Google’s Flu Trends team stopped publishing its own estimates: “Instead of maintaining our own website going forward, we’re now going to empower institutions who specialize in infectious disease research to use the data to build their own models.”
Key idea
The most dangerous failure may look normal
Crashes attract attention. Silent failures return plausible outputs with no obvious warning, allowing errors to scale through automated workflows.
Generative systems can sound fluent, ranking systems can quietly suppress alternatives, and risk models can shift who receives review. None of that arrives as an outage. Reliability design has to look for incorrect normality.
The COVID-19 chest-imaging models built during 2020 are the largest documented example. A systematic review led by Michael Roberts appeared in Nature Machine Intelligence on 15 March 2021. It searched papers and preprints from 1 January to 3 October 2020. It identified 2,212 studies and kept 415 after initial screening. Its conclusion: “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” Part of the reason is the control group. Several papers used a pneumonia image set as their non-COVID comparison. The reviewers note that it “consists of paediatric patients aged between one and five”. Such a model “is likely to overperform as it is merely detecting children vs. adults”. A parallel study by Alex DeGrave, Joseph Janizek and Su-In Lee found the same shape of problem from the other side. The systems “rely on confounding factors rather than medical pathology, creating an alarming situation in which the systems appear accurate, but fail when tested in new hospitals”. None of these models crashed. They reported high accuracy, in peer-reviewed papers, while measuring something other than the disease.
A graceful interface can conceal an ungraceful decision process.
Steps
A practical reliability review
Use this review before relying on an average metric or polished demo.
A public accident record shows what the stress-testing step is for. On 18 March 2018, in Tempe, Arizona, an automated test vehicle collided with a pedestrian. The US National Transportation Safety Board investigated. The automated driving system “first detected the pedestrian 5.6 seconds before impact”. It “never classified her as a pedestrian—or correctly predicted her path—because she was crossing N. Mill Avenue at a location without a crosswalk, and the system design did not include consideration for jaywalking pedestrians”. The ADS “changed the pedestrian’s classification several times, alternating between vehicle, bicycle, and other”. And “with each change in object classification, the ADS perceived the pedestrian as a new object without considering its location history”. Accumulated mileage had not surfaced any of this. By the time of the crash the same fleet “had traveled the route on which the crash occurred about 50,000 times”. The Board adopted its report on 19 November 2019. An out-of-scope input is not dangerous because it is rare. It is dangerous because nobody wrote it into the scope.
- 1
Define acceptable service
Specify quality, latency, availability, consistency, and failure-handling expectations for the workflow.
- 2
Segment the evidence
Measure performance across important classes, populations, environments, and difficulty levels.
- 3
Stress unusual conditions
Test missing data, adversarial inputs, novel combinations, peaks, and out-of-scope requests.
- 4
Test the fallback
Verify that users can recover when the model is unavailable, uncertain, or wrong.
- 5
Set monitoring triggers
Choose signals that prompt investigation, containment, rollback, or reevaluation.
- 6
Review real incidents
Study near misses and user corrections rather than waiting for aggregate metrics to move.
Reliability belongs to the service, not only the model
A reliable service combines model quality with input validation, calibrated policies, user guidance, monitoring, capacity planning, and recovery. The model is one term in that list. A weaker model with careful scope and a fallback will outperform a stronger one used without controls.
Reliability is also contextual. A system acceptable for suggesting photo tags may be unacceptable for denying benefits because the consequence, evidence, and remedy differ.
The acceptable error budget depends on what the system is allowed to do and how people can recover.
Key takeaways
- Reliability is defined by error patterns across relevant conditions, not only by one average metric.
- Model scores, confidence claims, and calibrated probabilities are different concepts.
- Edge cases and distribution shift can expose behavior that development data did not reveal.
- Silent plausible failures can scale more dangerously than obvious crashes.
- Graceful failure combines detection, limited action, fallback, logging, recovery, and later diagnosis.
- Reliability is a property of the whole service and must be judged relative to stakes and remedy.