AI literacy basics

From Model Output to Product Decision

Learn how AI outputs become recommendations or actions through thresholds, rules, business constraints, interfaces, human review, and fallback design.

By the end you can

Distinguish model scores from thresholds, decisions, and outcomes
Explain how product policy converts uncertain evidence into action
Recognize the trade-offs created by thresholds and review capacity
Design a basic decision policy with validation, fallback, and escalation

Example

A score does not decline a payment

Suppose a model assigns a transaction a fraud risk of 0.78. That number becomes consequential only when the surrounding system interprets it.

One bank may request a one-time code above 0.60 and block only above 0.90.
Another may send high-value transactions to a human analyst regardless of score.
A low-value merchant may accept more fraud risk to avoid rejecting legitimate customers.
A system may combine the score with location, device history, account status, and explicit compliance rules.
The final outcome also depends on whether the customer can appeal, verify identity, or complete the purchase another way.

The decision policy—not the raw score—determines who is interrupted, reviewed, approved, or denied.

Case

MiDAS: what happens when nothing interprets the score

When nothing interprets it, the score becomes the verdict. Between 1 October 2013 and 7 August 2015, Michigan’s Unemployment Insurance Agency let its Michigan Integrated Data Automated System (MiDAS) issue unemployment-fraud determinations on its own. The federal court that later heard the resulting claims recorded what that meant: “In the years material to this suit, the UIA largely did not employ any human review in making these automated determinations.” Of the 22,427 “robo-adjudications” that were reviewed, “over 93% did not involve fraud at all” (Cahoo v. SAS Institute Inc., No. 17-10657, E.D. Mich., 2 March 2018). The Michigan Supreme Court recorded the same result in Bauserman v. Unemployment Insurance Agency on 26 July 2022: a study conducted by the agency “concluded that, during this same period, approximately 93% of the automated system’s fraud determinations were incorrect”. The action bolted to that determination was Michigan’s quadruple penalty, the highest in the nation, collected through wage garnishment and seizure of tax refunds.

Comparison

Four layers that are often collapsed

Keeping these layers separate makes both evaluation and accountability clearer.

FigureComparison · 4 columns

Score or prediction

A model estimates a class, quantity, probability-like value, ranking, or candidate output.

Produced by inference
Contains model uncertainty and error
May require calibration
Example: 0.78 risk score

Threshold or policy

Rules translate model evidence and context into a decision category.

Reflects costs and capacity
Can vary by segment or value
May include legal constraints
Example: request verification above 0.60

Action

Software or a person blocks, approves, ranks, drafts, escalates, or requests more information.

Can be reversible or irreversible
May require human authorization
Needs a fallback
Example: hold for analyst review

Outcome

The real effect appears later in customer experience, safety, cost, rights, or organizational performance.

May be delayed or hard to observe
Can differ from the intended objective
Needs downstream measurement
Example: prevented fraud without excessive false blocks

The decision policy is part of the product design

A threshold expresses a trade-off. Raising it may reduce false alarms while allowing more harmful cases through; lowering it may catch more cases while increasing unnecessary interruptions.

The correct trade-off depends on stakes, review capacity, affected people, reversibility, and the cost of delay. No universal threshold exists. No number turns a model into a responsible decision-maker.

Thresholds encode values and operational constraints, not only mathematics.

Case

Results day 2020: a threshold with a country behind it

England’s 2020 exam grading shows a threshold carrying exactly that weight. With examinations cancelled, Ofqual standardised teacher-submitted centre assessment grades (CAGs) against each centre’s historical results — but only where the cohort was large enough for the regulator to trust the arithmetic. In Ofqual’s own account, “this approach would not be applied to very small groups of students (fewer than five candidates studying a subject within a centre) because the statistical basis for calculated grades was deemed to be too weak”, so those candidates kept their CAG unstandardised, while “for subject cohorts with five to 15 candidates, a taper was used which applied a linear weighting between the CAG outcomes and the calculated grades”. On results day, 13 August 2020, 39% of A level entries came out below the CAG and just over 2% above it. Four days later Ofqual withdrew the policy and awarded each candidate whichever of the two grades was higher. Nothing about the model changed in those four days; the rule about whose evidence counted did.

A stacked bar of all A level entries on results day 2020: 39% graded below the centre assessment grade, roughly 59% unchanged, just over 2% above. Beneath it, the cohort-size rule — fewer than five candidates keep the unstandardised grade, five to 15 get a linear taper, more than 15 are standardised. — A statistical standardisation that moved about 19 entries down for every one up, and a threshold on cohort size — not the model — deciding whose evidence counted. Ofqual, summer 2020; the unchanged share is derived as the balance.

Visual

A dependable path from input to action

A robust product creates opportunities to validate inputs, compare evidence, route uncertainty, and recover from failure.

FigureProcess · 6 steps

1
Validate input
Check format, permissions, completeness, and whether the case is within the intended scope.
2
Run model and controls
Produce the model output, apply deterministic rules, and attach relevant context.
3
Interpret uncertainty
Use thresholds, confidence information, disagreement, or out-of-scope detection to choose a route.
4
Select handoff
Inform a user, request review, execute a reversible action, or decline to automate.
5
Record the decision
Log version, evidence, policy, reviewer action, and reason codes where appropriate.
6
Observe the outcome
Measure what happened later and whether the policy should change.

One of these steps is written into United States credit law

One of these steps is a legal obligation in United States consumer credit, and it is drafted precisely against collapsing a score into a decision. The Equal Credit Opportunity Act Amendments of 1976 (Pub. L. 94-239, 23 March 1976) gave every applicant refused credit a right to reasons: “A statement of reasons meets the requirements of this section only if it contains the specific reasons for the adverse action taken” (15 U.S.C. §1691(d)(3)). Regulation B, which implements it, closes the obvious escape route. The statement “must be specific and indicate the principal reason(s) for the adverse action”, and “statements that the adverse action was based on the creditor’s internal standards or policies or that the applicant, joint applicant, or similar party failed to achieve a qualifying score on the creditor’s credit scoring system are insufficient” (12 CFR §1002.9(b)(2)). Falling below a cut-off is not a reason; it is the thing that still has to be explained.

Key idea

Automation can hide policy rather than remove it

When an AI system decides quickly, users may read the result as objective, or as inevitable. It is neither. Someone chose the target, the training evidence, the threshold, the exception rules, and the appeal process.

Hiding those choices behind a score weakens accountability. A well-designed system makes consequential policies explicit enough to review, test, and challenge.

European law has gone one step further. It has treated the score itself as the decision when everything downstream follows from it. The Court of Justice of the European Union decided Case C-634/21, SCHUFA Holding (Scoring), on 7 December 2023. It ruled on Article 22(1) of the GDPR, which covers automated individual decision-making. A credit information agency establishes automatically “a probability value based on personal data relating to a person and concerning his or her ability to meet payment commitments in the future”. That establishment is itself “automated individual decision-making”, the Court held. The ruling applies “where a third party, to which that probability value is transmitted, draws strongly on that probability value to establish, implement or terminate a contractual relationship with that person”. The scoring agency could no longer answer that it had merely supplied a number. The bank, on that argument, had made the decision.

Automated decisions still contain human choices; automation changes where those choices are located.

Position

Whoever sets the threshold is writing the policy

The score is arithmetic. The threshold is a judgement about who absorbs the errors, and it is usually made by whoever sits closest to the code rather than by whoever answers for the outcome. That gap is where the failures in this lesson live. Michigan ran an automated fraud determination for nearly two years in which the number was the verdict. In 2020 an examinations regulator standardised grades against each centre's history, and a cohort-size threshold inside that rule decided whose grades were standardised at all and whose were left as submitted.

So when you review a system, do not start with the model. Ask who chose the cut-off, what they were optimising when they chose it, who else was in the room, and what the review capacity was on the day it went live. If nobody can name the person, you are looking at policy that no one signed.

A threshold is a political decision written in a configuration file.

Steps

Draft a decision policy before deploying the model

A short policy forces the team to connect model evidence with consequences.

FigureProcess · 6 steps

1
Define decision bands
Specify which outputs are safe for automatic handling, require review, or must be rejected as out of scope.
2
State the cost trade-off
Describe the relative harm of false positives, false negatives, delay, and inconsistent treatment.
3
Protect edge cases
Create rules for missing inputs, novel cases, vulnerable users, and high-impact exceptions.
4
Design appeal and override
Allow authorized people to correct or contest decisions, and record why they did so.
5
Set capacity limits
Ensure queues, staffing, and latency targets make the review policy operationally possible.
6
Define change triggers
Name the evidence that would justify adjusting thresholds, rules, or model versions.

Analogy

A test result is not yet a decision

A laboratory test returns a value outside a reference range. The number can inform care, but a clinician considers symptoms, history, test quality, and the consequences of acting.

A model score similarly contributes evidence rather than issuing meaning by itself. Established laboratory tests, though, come with measurement standards, a causal interpretation, and clinical validation. Many AI scores come with none of the three.

The reference range is itself a policy. Under the CLSI and IFCC convention, laboratories place the lower and upper reference limits at the 2.5th and 97.5th percentiles. The results come from a healthy reference population of at least 120 people. Nathan Timbrell’s 2024 review in the British Journal of Biomedical Science states the consequence: “this means that 5% of this disease-free population falls outside of the RI, with 2.5% above the upper reference limit (URL) and 2.5% below the lower reference limit (LRL)”. Run twenty such tests on one healthy person. If the results were independent, the chance that at least one falls outside its range is about 64%. The clinician’s judgement is not politeness added to the number. It is what keeps a percentile convention from being read as a diagnosis.

Evidence becomes a decision only through a policy suited to the context.

Model metrics and product metrics answer different questions

A model metric may measure precision, recall, error, or ranking quality. A product metric may measure time saved, complaints, appeal success, incident severity, user comprehension, or downstream outcomes.

Both levels matter. Improving the model is valuable only when the system translates that improvement into a better workflow without creating new harms elsewhere.

Netflix published that conclusion about its own most celebrated model win. Xavier Amatriain and Justin Basilico described the Netflix Prize in April 2012. The company “had to come up with a proxy question that was easier to evaluate and quantify: the root mean squared error (RMSE) of the predicted rating”. The finish line was set. It meant cutting the RMSE of its Cinematch system from 0.9525 to 0.8572. Netflix took the two best-performing algorithms out of the first Progress Prize ensemble. It put them into production. Those two were matrix factorisation and a restricted Boltzmann machine. The ensemble of hundreds of models that finally crossed the line never shipped. The team “evaluated some of the new methods offline”. The additional accuracy gains they measured “did not seem to justify the engineering effort needed to bring them into a production environment.” The proxy metric had been beaten, and the company’s focus “had shifted to the next level by then”.

A model win that does not improve the decision system is not yet a product win.

Key takeaways

A model score becomes consequential only through thresholds, policies, interfaces, and human or automated actions.
Score, decision policy, action, and outcome must be analyzed as separate layers.
Thresholds encode trade-offs among false positives, false negatives, delay, cost, and review capacity.
Consequential policies should expose scope limits, fallback, override, appeal, and change triggers.
Automation relocates human choices; it does not remove values or accountability from a decision.
Model metrics must be connected to product and outcome metrics before an improvement can be called valuable.