Supervised Learning: Learning From Labeled Examples

Supervised learning trains a model on examples that pair an input with a target output. That target may be a class, a number, an ordering, a sequence, a mask, or another structured object.

The learner searches for a relationship that performs well on future-like cases. It does not merely memorize the table. Nor does the label guarantee direct access to truth.

The best-documented answer key in medical imaging shows what that reservation is worth. The training target was 128,175 retinal fundus photographs, each graded 3 to 7 times by a panel of 54 US-licensed ophthalmologists and ophthalmology senior residents. Gulshan and colleagues published the result in JAMA in 2016: AUC 0.991 on EyePACS-1 (9,963 images) and 0.990 on Messidor-2 (1,748 images). A later study had a separate clinical validation set graded twice over — once by the majority decision of three ophthalmologists, once by the face-to-face adjudicated consensus of three retinal specialists. The two answer keys disagreed 193 times, 36 of those by at least two grades. Krause and colleagues counted why: “The most common causes were missed MAs (36%), artifacts (20%), and disagreement regarding whether a lesion was an MA or a hemorrhage (16%).”

The scores moved with the key. Measured against the adjudicated consensus, the original Gulshan model scored AUC 0.942 for moderate-or-worse disease. Measured against the retinal specialists' majority decision, the same model scored 0.952. A retrained higher-resolution model, tuned on adjudicated grades, reached 0.986. Not one photograph changed. Only the process that decided what the photographs meant.

The supervised workflow converts operational history into examples. It fits a mapping. It then evaluates that mapping on data excluded from fitting.

Five stages carry it. Construct the examples, by choosing the unit, the prediction time, the input features and the target maturity. Fit a model, adjusting parameters to reduce a loss that compares predictions with targets. Validate choices, using separate evidence to select features, capacity, thresholds and training settings. Test the claim on an untouched set that reflects the intended use. Then connect the output to action — decisions, review, abstention, or prioritization.

Each of the failures in this lesson was introduced at one of those five stages, not by the choice of algorithm.

The form of the target determines the immediate prediction task. The more structure the target carries, the more the answer key costs.

Microsoft COCO, published in 2014, contains 2.5 million labeled instances with per-instance segmentations, across 328,000 images in 91 object categories. Cityscapes provides 5,000 images with high-quality pixel-level annotations, plus 20,000 coarsely annotated images from street scenes in 50 cities. Cordts and colleagues state the price per image in the 2016 paper: “Annotation and quality control required more than 1.5 h on average for a single image.”

A pixel-level target is not a different setting on the same machine. It is a different budget. And every one of those hours is a human decision that the next annotator can make differently.

Convenience decides many targets. Supervised projects often use one that is easier to record than the outcome stakeholders truly value, and the substitution is rarely announced to the people it lands on.

The largest measured case is a commercial risk-prediction algorithm applied, by industry estimates, to roughly 200 million people in the United States each year. Its job was to find the patients who most needed extra care. Its target was not illness. It was health-care cost — recorded for everyone, available immediately, and correlated with need. Obermeyer and colleagues measured the price of that substitution in Science on 25 October 2019. Removing the resulting gap would raise the share of Black patients automatically identified for high-risk care management, at the 97th-percentile threshold, from 17.7% to 46.5%. Their abstract closes on the general point: “We suggest that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.”

The same day, the New York State Department of Financial Services and Department of Health wrote jointly to David S. Wichmann, CEO of UnitedHealth Group, demanding that the company justify or stop using Impact Pro. The gap between a target and an outcome is not an academic distinction. It took days to become a regulator's letter to a named executive.

Three kinds of label sit behind that story. A direct target records the outcome itself, such as a confirmed component failure. It is usually slower or costlier to obtain, comparatively clear to interpret, and still vulnerable to measurement error. A proxy target stands in for a harder outcome, as cost stood in for need. It is often abundant and immediate, it can reward superficial behavior, and it needs product-level validation before anyone trusts it. A policy-created label exists only where a prior decision process looked — fraud confirmed only after an investigation that a rule triggered in the first place. Its coverage depends on historical policy, it can reproduce earlier blind spots, and it needs selective-label analysis. The third kind is the one that looks most like ground truth and is least like it.

This step comes first. A short design review can expose problems that no algorithm choice will solve. Fix the prediction moment, by stating exactly what information exists when the output must be produced. Define the target, including how it is observed, disputed, delayed, or revised. Map the population, identifying which entities and conditions enter the dataset and which are missing. Set the action for positive, negative, uncertain, or missing predictions. Reserve evaluation evidence that will not influence model or threshold selection.

Two of those five are no longer only good practice in the United States. In January 2025 the US Food and Drug Administration issued draft guidance on AI-enabled device software functions, under Docket No. FDA-2024-D-4488, with comments closing 7 April 2025. It first says what an answer key is: “For the purposes of this guidance, a reference standard is the best available representative truth that can be used to define the true condition for each patient/case/record.” It then requires sponsors to document how the reference standard was established, the uncertainty inherent in it, how equivocal or missing reference-standard results were handled, and how independence of test data from training data was ensured.

Step two of the review, defining the target, and step five, reserving evaluation evidence, are those same demands — with a docket number attached, for anyone shipping a supervised medical device.

Supervised learning fits when historical or newly collected targets meaningfully represent the desired behavior, and future inputs resemble the evaluated population enough for prediction.

It becomes fragile on the label side. Labels may be unavailable, strategically manipulated, policy-dependent, severely delayed, or disconnected from the true objective. Those conditions motivate the approaches developed later in this path.

Google Flu Trends is the counted example. Its targets were sound — the CDC's influenza-like-illness reports — and its inputs were search queries, the model fitted by matching 50 million search terms to 1,152 data points. It ran systematically high. Lazer and colleagues reported that in Science in 2014, and their figure caption states it flatly: “From 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence 100 out of 108 weeks.” In February 2013 it predicted more than double the CDC proportion of ILI doctor visits. A separate reassessment of the same system, at three geographic scales, had already appeared in PLOS Computational Biology in 2013. Nothing was wrong with the answer key. The relationship between the inputs and it had moved, and 1,152 data points were never enough to notice.