Kinds of learning

Active Learning and Human-in-the-Loop Systems

Learn how query strategies select examples for annotation, why uncertainty is not enough, and how to design sustainable human review and escalation.

By the end you can

Define active learning as selective acquisition of labels
Compare uncertainty, diversity, and expected-impact query strategies
Recognize sampling bias and reviewer-burden risks
Design human-in-the-loop workflows that preserve authority and feedback quality

The scarce resource is often expert judgment

Unlabeled examples are abundant. Annotation is expensive. Labeling at random spends expert time on repetitive, easy cases, so active learning instead selects the examples expected to improve the model or clarify a decision boundary.

It helps to know what that time actually costs. One pathologist, reading 129 whole-slide images with no time constraint, took approximately 30 hours. That is about 14 minutes of specialist attention for a single slide. The slides came from CAMELYON16 — 400 whole-slide images from Radboud University Medical Center and University Medical Center Utrecht, a challenge that closed in November 2016. Fourteen minutes is the unit an active-learning loop is spending every time it adds a case to the queue.

The method changes the annotation distribution. A model-chosen sample is useful for learning. It may not represent natural prevalence, and it should not replace an unbiased evaluation set.

Comparison

Three reasons to ask for a label

A good query batch balances uncertainty, coverage, cost, and operational importance.

FigureComparison · 3 columns

Uncertainty sampling

Request labels where the current model is least certain.

Targets decision boundaries
Simple to implement
Can chase outliers or noise
Depends on meaningful uncertainty

Diversity sampling

Choose examples that cover different regions of the pool.

Reduces redundant queries
Improves broad coverage
May select easy examples
Depends on representation quality

Expected impact

Estimate which labels could improve a metric or decision most.

Connects queries to value
Can include annotation cost
Computationally harder
Relies on model assumptions

Example

129 slides, 30 hours, and what a time limit costs

CAMELYON16 priced expert attention in public. Thirty-two algorithms from 23 teams were scored on the same 129-slide test set, and the best reached an AUC of 0.994 (95% CI, 0.983–0.999). Eleven pathologists read those slides under a two-hour simulated time constraint and averaged 0.810, with a range of 0.738 to 0.884. One pathologist read them with no time constraint and reached 0.966 (95% CI, 0.927–0.998). JAMA published the comparison in 2017, together with the price of that last figure: “The pathologist WOTC required approximately 30 hours for assessing 129 whole-slide images.”

Both halves matter to a loop designer. Unhurried expert reading comes close to the best algorithm, and it costs about 14 minutes a slide. Compress the same reading into two hours and accuracy falls from 0.966 to 0.810. The queue policy is not choosing between an expert and no expert. It is choosing how much of that 14 minutes each queried case gets.

The initial labeled set covers common tissue and known disease patterns; CAMELYON16 released 400 whole-slide images to build one from.
Uncertainty sampling proposes ambiguous boundaries and unfamiliar artifacts — precisely the cases where the drop from 0.966 to 0.810 is paid.
Diversity constraints prevent one scanner, one patient or one centre from dominating a batch; the CAMELYON16 slides came from two centres, Radboud University Medical Center and University Medical Center Utrecht.
Reviewers can mark poor image quality instead of forcing a diagnosis.
Annotation time and disagreement are logged as signals of case difficulty: 30 hours for 129 slides is a measurement, not an impression.
A separate representative test set remains untouched by the query policy — the 129-slide set here scored 32 algorithms from 23 teams and the 11 time-constrained pathologists on identical material.

Case

115 pathologists, 240 biopsies, and 48% agreement on atypia

The label the queue is asking for is not always stable. Give 240 breast biopsy cases to 115 practising pathologists and you get 6,900 interpretations. Compared against a consensus reference diagnosis, 75.3% of them matched — 5,194 of 6,900. JAMA published that in 2015, and the agreement was not evenly spread. Invasive carcinoma: 96%. Benign without atypia: 87%. Ductal carcinoma in situ: 84%. Atypia: 48%. On the hardest category the field agreed with the reference less than half the time. An active-learning loop that routes its most uncertain cases to one annotator is, on a category like that, sampling a single opinion and recording it as ground truth.

A zero-based bar chart of how often 115 pathologists agreed with a consensus reference diagnosis on 240 breast biopsy cases, with each category’s 95% confidence interval drawn beneath its bar. Invasive carcinoma reaches 96% (94–97), benign without atypia 87% (85–89), ductal carcinoma in situ 84% (82–86) and atypia 48% (44–52); the overall rate, 75.3%, is marked as a dashed reference line. A label bought from this pool is near-certain for invasive carcinoma and close to a coin-flip for atypia — and an active-learning loop that queries the hard cases buys mostly the second kind. The study produced 6,900 interpretations, of which 5,194 matched the reference and 1,706 did not; 5,194 of 6,900 reproduces the reported 75.3% (95% CI 73.4–77.0%). — Before measuring a model against expert labels, measure the experts: concordance here ranges from 96% to 48% depending on the diagnosis.

Visual

A human-in-the-loop system has several feedback roles

Human involvement can occur before, during, or after prediction, and each role needs authority and interface design.

FigureHierarchy · 5 levels

Annotation
People create targets used for future training.
- Review
  People inspect model outputs before an action occurs.
  - Escalation
    People handle uncertain, novel, or high-risk cases.
    - Intervention
      People override or stop an automated policy during operation.
      - Audit
        People examine patterns, slices, and system behavior after decisions.

Key idea

The most uncertain case is not always the most useful case

A model may be uncertain because the input is corrupted, outside scope, or inherently ambiguous. Labeling such cases repeatedly can consume experts without improving the intended boundary.

Combine uncertainty with diversity, data-quality filters, domain value, and annotation cost. Measure the marginal gain from each query strategy.

The score doing the selecting has itself been audited. Guo and colleagues opened a 2017 calibration paper with the result: “We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated.” Expected calibration error for an uncalibrated ResNet-110 on CIFAR-100 was 16.53%. A 5-layer LeNet on the same data: 4.85%. ResNet-152 on ImageNet: 5.48%. Temperature scaling cut the CIFAR-100 ResNet-110 figure to 1.26%. A queue ranked by a confidence off by that much is ranking partly on difficulty and partly on the model's own miscalibration. The repair was recalibration, not a new acquisition function.

The marginal gain from active learning itself has also been measured, and it is thin. Lowell and colleagues swept acquisition functions, models and datasets in 2019, across text classification and named-entity recognition. In 75% of model, dataset and training-size combinations, some acquisition function beat random i.i.d. sampling. But a practitioner has to commit to one function in advance. Any single strategy beat random “in only a slight majority (60.9%) of cases”.

The second finding is the one that costs money later. An actively acquired dataset is coupled to the model that acquired it. Where the successor model differed from the acquisition model, “only 37.5% of the tabulated data points … outperform the i.i.d. baseline”. Labels bought with this model's uncertainty may be worth less to the next model than labels drawn at random.

Query difficulty and learning value are related, but they are not identical.

Analogy

Choosing cases for a specialist conference

Time at a hospital conference is limited. Clinicians select cases that are uncertain, educational, or likely to change treatment, while routine cases remain in ordinary workflow.

Active learning resembles this allocation of expertise. Clinicians know which cases puzzle them. Model uncertainty is not that knowledge — it can be poorly calibrated, and it may skip the groups the model already misrepresents confidently.

How badly it is calibrated, though, is a property of the model in front of you and not a standing law. Minderer and colleagues re-ran the question on newer models in 2021 and found “the most recent models, notably those not using convolutions, are among the best calibrated”. The query policy therefore inherits an empirical question rather than a permanent defect. Before a specialist's day is allocated by a confidence score, somebody has to have measured that score on this model and this data.

Expert time should target informative and consequential gaps, not uncertainty alone.

Steps

Design a sustainable annotation and review loop

The loop must serve both model improvement and human work quality. Two of its six steps have already been measured or legislated by somebody outside the team. Neither survives being treated as a formality.

Step 2 reserves evaluation data, which does not make that data true. The held-out sets everyone scores against have been counted: “Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set.” That is Northcutt and colleagues, 2021. Beyer and colleagues re-annotated the same ImageNet validation set independently and found “the original ImageNet labels to no longer be the best predictors of this independently-collected set”. Two groups, one conclusion. The held-out set is a second annotation job with an error rate of its own, and it deserves the same audit as the training queue.

Step 4 designs the interface, and in one domain the interface design is the legal test. US law keeps clinical decision support software outside device regulation only where it is intended for “enabling such health care professional to independently review the basis for such recommendations that such software presents so that it is not the intent that such health care professional rely primarily on any of such recommendations”. That is the fourth criterion, written into the Federal Food, Drug, and Cosmetic Act by the 21st Century Cures Act in 2016. FDA's Clinical Decision Support Software guidance, issued 29 January 2026, reads it as a workload test and not only a transparency test: “FDA does not consider software functions intended for a critical, time-sensitive task or decision to meet Criterion 4, because an HCP is unlikely to have sufficient time to independently review the basis of the recommendations.” Showing the basis is not enough if the clock does not allow it to be read.

FigureProcess · 6 steps

1. Define the query goal
Choose boundary learning, rare-case coverage, error discovery, or decision protection.
2. Reserve evaluation data
Keep representative labeled evidence outside the active pool.
3. Balance the batch
Combine uncertainty, diversity, importance, and reviewer cost.
4. Design the interface
Show necessary context, allow abstention, and capture rationale selectively.
5. Monitor burden
Track time, disagreement, fatigue, and repeated exposure to harmful content.
6. Recheck blind spots
Sample cases the model considers easy to detect confident systematic error.

“Human in the loop” is not a safety guarantee

A reviewer may be rushed. They may also overtrust the system, lack authority, or see insufficient evidence, and automation bias can turn nominal oversight into a rubber stamp. That sentence is now binding law. The Artificial Intelligence Act — Regulation (EU) 2024/1689, published in the Official Journal of the European Union on 12 July 2024 — requires in Article 14(4) that the people assigned oversight of a high-risk AI system be able to understand its capacities and limitations, to disregard, override or reverse its output, and to interrupt it via a 'stop' button. Article 14(4)(b) names the failure mode. They must be enabled “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons;”. For remote biometric identification, Article 14(5) does not trust a single reviewer at all: no action may follow an identification “unless that identification has been separately verified and confirmed by at least two natural persons with the necessary competence, training and authority”.

What that text is written against has been recorded twice in the US federal file, at deployment scale. A Model X crashed in Mountain View on 23 March 2018. The National Transportation Safety Board's report, adopted on 25 February 2020, gives a probable cause: “Contributing to the crash was the Tesla vehicle’s ineffective monitoring of driver engagement, which facilitated the driver’s complacency and inattentiveness.” The Board also concluded that steering-wheel torque is “an ineffective surrogate measure of driver engagement”. The loop was measuring hands, not attention.

On 12 December 2023 Tesla filed NHTSA recall 23V-838, covering 2,031,220 vehicles, and stated that “the prominence and scope of the feature's controls may not be sufficient to prevent driver misuse”. NHTSA closed its investigation EA22002 on 25 April 2024 with 956 crashes reviewed, 211 frontal-plane crashes and 14 deaths. It also reported one number about the supervising humans: across the 135 incidents where data existed, drivers “either did not brake or braked less than one second prior to the crash in 82 percent of the incidents”.

So specify who can override, how long review takes, what context is shown, and what happens after disagreement — the same four things the AI Act obliges and FDA's fourth criterion tests. Human involvement must be designed and evaluated like any other component. At two million deployments, the supervising human did not brake at all in 82 percent of the crashes they were the control against.

Case

The reviewers shown a risk score, and the 64.1% who did worse than it

The rubber stamp has been measured. In 2019 Green and Chen paid 554 participants on Amazon Mechanical Turk to predict pretrial recidivism for the same 500 defendants, showing half of them a risk score of accuracy comparable to COMPAS. Seeing the score helped. It was still not enough. The treatment group “achieved a 2.6% lower average reward and a 46.5% higher false positive rate than the risk assessment”, and only 23.7% of them beat the tool they were reading, against “64.1% who earned a lower reward than the risk assessment”.

The finding that matters most for oversight design is that they could not tell. “Participants’ confidence in their performance was negatively associated with their actual performance”, and their judgments of the tool's accuracy and fairness “had no association with the risk assessment’s actual accuracy and fairness”. A reviewer who cannot rank themselves against the model cannot be the control on it.

Position

Both ends of the loop are people, and neither end has a published error rate

A model that shipped without an evaluation would be indefensible. A review step that ships without one is ordinary. Both of the people in this loop have in fact been measured, with results no product document quotes. At the labelling end, 115 pathologists reading 240 breast biopsy cases agreed with the reference diagnosis on 75.3% of 6,900 interpretations — and on 48% of the atypia cases. That is a coin toss on the category a loop built around uncertainty will keep surfacing. The same profession, given 129 whole slides and no clock, spent about 14 minutes each and reached an AUC of 0.966. Given two hours for the set: 0.810.

At the review end the finding is worse than imperfect. Green and Chen's participants read a risk score, which helped them, and still finished 2.6% below the score's own average reward with a 46.5% higher false positive rate. Of them, 23.7% beat the tool they were reading and 64.1% did worse than it. Their confidence was negatively associated with their actual performance. Their judgments of the tool's accuracy and fairness had no association with its actual accuracy and fairness. Those were 554 Mechanical Turk participants and not judges, so the percentages do not transfer to a trained reviewer. The mechanism does, and the mechanism is what the design is resting on.

There is a third set of people nobody counts: whoever annotated the evaluation data both ends are scored against. The ImageNet validation set holds 50,000 images and 2,916 label errors — 5.83% — with Mechanical Turk workers confirming 51% of the algorithmically flagged candidates. A loop whose reference standard carries an unmeasured error rate cannot report anyone else's. Someone who cannot tell whether they are improving on the model cannot be the control on it. Measure the loop the way you measure the model, or stop describing it as oversight.

“An expert labels it, an expert reviews it” is a diagram. It becomes a safeguard when somebody reports the expert's numbers.

Example

Measure the human system as carefully as the model

Reviewer quality and sustainability determine whether an active loop can continue, and every line below is a quantity somebody has already published for some domain.

Decision time by case type and information completeness — 30 hours for 129 whole-slide images is what an unhurried reading of one queue actually cost.
Agreement and adjudication rate for queried examples — 48% concordance on atypia is what an unadjudicated queue would have recorded as ground truth.
Abstention rate when evidence is insufficient.
Emotional or physical burden for harmful-content and high-stakes review.
Override and appeal outcomes after model-assisted decisions — Article 14(4) of the AI Act requires that overriding be possible; whether it is ever used is a number you have to collect.
Coverage of languages, regions, rare classes, and confident cases — including the reference set itself, whose own labels ran at least 3.3% wrong across ten standard benchmarks.

Key takeaways

Active learning chooses which examples deserve scarce annotation effort, and that effort is measurable: about 14 minutes of unhurried specialist time per whole-slide image in CAMELYON16.
Uncertainty, diversity, expected impact, and annotation cost support different query goals. In the 2019 sweep by Lowell and colleagues, any single acquisition function beat random i.i.d. sampling in only 60.9% of cases.
Actively selected data should not replace a representative evaluation sample — and the reserved sample is not ground truth either, at 2,916 label errors in the 50,000-image ImageNet validation set.
Human roles include annotation, review, escalation, intervention, and audit. Article 14 of the AI Act, Regulation (EU) 2024/1689, turns several of them into obligations for high-risk systems, including two-person confirmation for remote biometric identification.
Reviewer burden, interface design, abstention, and authority determine whether oversight is meaningful. FDA treats a critical, time-sensitive decision as one an HCP is unlikely to have time to review independently.
Confident-case audits protect against blind spots created by the query policy, because the confidence itself can be wrong: 16.53% expected calibration error for an uncalibrated ResNet-110 on CIFAR-100, cut to 1.26% by temperature scaling.