Skip to content
AI.info

How machines learn

Feedback Loops and Selective Labels

Understand how recommendations, ranking, review policies, and interventions alter future behavior and which labels become visible.

By the end you can

Visual

Equal looks, unequal clicks: the model helps create the behavior it later learns from

A recommendation system does not merely observe preference. It chooses exposure, which affects clicks, watch time, purchases, and future training data. That the click reflects position as well as preference is not an inference — it was measured with an eye-tracker.

Users looked at the abstracts ranked 1 and 2 with almost equal frequency. They clicked the first one far more often. That was 2005, across 34 recruited subjects, 29 of them with usable gaze data. Joachims and his co-authors found the result worth remarking on: “It is very interesting that users click substantially more often on the first than on the second link, while they view the corresponding abstract with almost equal frequency.”

A second phase tested whether relevance rescues the signal. It ran 22 recruited subjects, 16 usable: 6 “normal”, 5 “swapped”, 5 “reversed”. In the unmanipulated normal condition, counting only the sessions where a user clicked exactly one of the top two, the top result was clicked in 19 of the 20 cases in which it was the more relevant of the pair. Where the second result was the more relevant, it was clicked in only 2 of 7. Then a proxy secretly swapped the top two: “there is still a strong bias to click on link one even if the second abstract is more relevant.”

Microsoft Research reached the same conclusion three years later by a different route. In 2008 it reported that “the probability of click is influenced by a document's position in the results page”. Rank one is therefore partly a self-fulfilling prophecy. The ranking the model produced yesterday is inside the click it trains on tomorrow.

FigureProcess · 5 steps
  1. 1

    Model ranks items

    Past data and objectives determine what is shown.

  2. 2

    Users receive selective exposure

    Unshown items cannot receive ordinary feedback.

  3. 3

    Behavior is recorded

    Clicks and dwell time reflect preference, position, interface, and availability.

  4. 4

    Training data is updated

    Observed interactions favor previously exposed items.

  5. 5

    Next model reinforces patterns

    Popularity and blind spots can grow unless the loop is managed.

Comparison

Feedback can amplify, counteract, or distort

The effect depends on how predictions change the process. Each of the three shapes has a worked instance later in this lesson.

A reinforcing loop sends patrols back to the neighbourhoods they already policed — PredPol on Oakland records. A balancing loop is the sepsis alert or the maintenance visit that removes the very event the score predicted. The score then appears to fail in proportion to how well the intervention works. A measurement loop is judicial bail: the outcome exists only for the defendants a judge released, so the metric describes the policy's coverage as much as the model's skill.

FigureComparison · 3 columns

Reinforcing loop

Model actions increase the future prevalence of patterns they favor.

  • Popularity can compound
  • Selected groups receive more opportunity
  • Errors can self-confirm
  • Example: recommended creators gain more engagement

Balancing loop

Actions reduce the condition the model predicts.

  • Successful intervention makes positives disappear
  • Naive evaluation may punish helpful models
  • Counterfactual outcomes are missing
  • Example: maintenance prevents failure

Measurement loop

The policy changes which cases receive labels or attention.

  • Reviewed cases are not representative
  • Unselected outcomes stay unknown
  • Metrics reflect policy coverage
  • Example: only high-risk transactions are investigated

Case

PredPol on Oakland's records: twice the patrols, equal drug use

PredPol's published algorithm, run on Oakland police records of drug crimes, would have targeted black people at roughly twice the rate of white people. Estimates from public-health surveys put illicit drug use in Oakland at roughly equal rates across racial and income groups. Lum and Isaac ran that experiment in Significance in October 2016.

The mechanism was modelled two years later, by five researchers at the first FAT* conference. In a system updated on the crime its own patrols discover, “police are repeatedly sent back to the same neighborhoods regardless of the true crime rate”. Incidents reported by residents “can attenuate the degree of runaway feedback” but “cannot entirely remove it”. The records then measure where the patrols went, not where the crime was.

You observe outcomes for decisions that were made

A lender observes repayment for approved applicants, not for rejected ones. A hiring system observes job performance for hired candidates. A fraud team verifies cases sent to investigation.

The label dataset is therefore conditional on previous decisions. Training a model as if unobserved cases were ordinary negatives can reproduce the old policy and hide missed opportunities or harms.

Missing outcomes are often caused by decision policy, not random chance.

Case

Bail: an outcome recorded only for the defendants a judge released

The term is not informal. Five researchers named “the selective labels problem” at KDD in 2017, and judicial bail is the worked case. Whether a defendant fails to appear in court is observed only for the defendants some judge chose to release. For the detained there is no outcome at all.

Their proposed answer is contraction. It does not try to impute the missing outcomes. It compares model and judge “without resorting to counterfactual inference”. It harnesses instead the heterogeneity of the judges themselves. It leans on the most lenient of them, whose released set already contains many of the cases stricter judges detained.

Predict first

A hundred machines, serviced in time

A factory uses a model to predict which of its machines will fail within the month. It flags 100 of them. The maintenance crew services all 100 that week, as the model’s owners intended. At the end of the month, 10 of the 100 have failed anyway.

Before you read on, predict what precision a dashboard will compute for the model from this record. Then say what else you would need to know to judge whether the model was any good.

Settle on an answer, then read on — the next section is the reveal

Example

What the record shows, and what happened

The numbers below are invented, to go with the hundred machines above; the mechanism is the one the callout below describes.

  • What would have happened. Left alone, 60 of the 100 flagged machines would have failed that month. The model was right about 60 of 100: a precision of 60%.
  • What the service did. It saved 50 of those 60. Ten failed anyway.
  • What the record says. 100 flags, 10 failures: a precision of 10%. The better the crew works, the lower that number goes, although the model has not changed.
  • What retraining on it would learn. 90 of the flagged machines are now labelled “did not fail”. A model refitted on those labels learns that machines like them are safe, flags fewer of them, and fewer get serviced. That is the loop a 2021 paper, described below, formalises: scores drifting toward predicting their own effect.
  • What would have told the truth. A log of each prediction and each service, and, where it is safe, a small random share of flagged machines whose service is delayed a few days, so that some outcomes are seen without the intervention.

The model was right six times in ten; the record said once in ten, because the repair erased the evidence.

Key idea

A good model can make its own predictions look wrong — and retraining makes it worse

If a model predicts machine failure and maintenance prevents it, the recorded outcome is “no failure.” Treating this as a false positive ignores the intervention that changed the result.

This is a named and published failure mode, not an intuition. A 2019 paper in the Journal of the American Medical Informatics Association is titled Prognostic models will be victims of their own success, unless… Its abstract states the paradox in a sentence: “The more effective a model and intervention(s) are at improving outcomes, the faster a model will appear to degrade.” The better the response to the alert, the worse the alert looks in the record.

Refitting on those post-intervention labels does not repair the distortion. It compounds it. A 2021 AISTATS paper formalises the existing score as a causal pathway into the outcome. Naive repeated updating, it shows, can drive successive scores toward a point where “they predict their own effect”, or into a stable oscillation between two values — and “even if model-fitting procedures improve, actual performance may worsen”.

Log the prediction, action, timing, and intervention. Evaluation may need to compare policies, use controlled experiments, or define outcomes that account for successful prevention.

Prediction quality and intervention effect cannot always be separated from observational labels.

Some randomization preserves evidence: Twitter's 1% who never got the algorithm

Small controlled exploration can expose items, inspect cases, or choose actions outside the current policy. This creates data about alternatives and reduces blind spots.

One platform kept such a holdout running for years. When Twitter introduced machine-learning personalisation of the Home timeline in 2016, it permanently excluded a randomly chosen 1% of global users. They kept a reverse-chronological feed. The PNAS paper that used them, in PNAS's issue of 4 January 2022, states the design in its abstract: “We provide quantitative evidence from a long-running, massive-scale randomized experiment on the Twitter platform that committed a randomized control group including nearly 2 million daily active accounts to a reverse-chronological content feed free of algorithmic personalization.”

What that group made measurable, the logs of the personalised feed could not have shown, because there is nothing in them to compare against. Using the holdout, the authors found that in 6 of the 7 countries studied the mainstream political right was algorithmically amplified more than the mainstream left. In the US media landscape, amplification favoured right-leaning news sources.

Outsiders without such a holdout have to build the comparison themselves. An independent Berkeley-led team did so for 806 Twitter users between 11 and 27 February 2023, collecting for each participant both the tweets the engagement-based algorithm would have shown and the most recent tweets from the accounts they followed. Relative to that reverse-chronological baseline, engagement-based ranking amplified anger by 0.47 SD, partisanship by 0.24 SD and out-group animosity by 0.24 SD. Twitter's holdout was built in; this one had to be reconstructed, one participant at a time.

Exploration carries cost and risk. High-stakes domains require ethical review, bounded options, informed consent where applicable, and safer designs such as expert adjudication, retrospective labeling, or staged pilots.

Analogy

An analogy: judging dishes only by what the waiter recommends

A restaurant measures demand while its waiters recommend the same three dishes. Those dishes appear popular. The unmentioned dishes receive little evidence about customer preference either way. The waiter's spoken list is the ranking. The secretly swapped abstracts in the eye-tracking study are what happens when someone quietly reorders it and the orders follow the position rather than the food.

Recommendation and decision systems create similar exposure bias. Algorithmic policies personalize exposure, affect long-term behavior, and operate at a scale no dining room reaches.

Steps

Design observability before deploying the policy — in the EU, it is Article 15(4)

Feedback loops become easier to study when the system records the choices that generated the data: the choice set, the policy context, the interventions, a protected slice of exploration data, long-term effects, and safeguards on self-reinforcing behaviour.

For one class of system the last step is not craft advice. It is law, with an article number. Regulation (EU) 2024/1689 of 13 June 2024, the Artificial Intelligence Act, states in Article 15(4): “High-risk AI systems that continue to learn after being placed on the market or put into service shall be developed in such a way as to eliminate or reduce as far as possible the risk of possibly biased outputs influencing input for future operations (feedback loops), and as to ensure that any such feedback loops are duly addressed with appropriate mitigation measures.”

The same obligation reaches back into the data pipeline. Recital 67 asks data-governance practices to attend to bias 'especially where data outputs influence inputs for future operations (feedback loops)'. For a continuously-learning high-risk system in the EU, "manage the loop" is a documented duty attached to a numbered provision. Under the Digital Omnibus, Regulation (EU) 2026/1744, it applies from 2 December 2027 for the high-risk uses listed in Annex III, and from 2 August 2028 for AI built into the products listed in Annex I.

FigureProcess · 6 steps

  1. 1. Log the choice set

    Record which options were available, shown, hidden, or infeasible.

  2. 2. Log policy context

    Store model version, scores, threshold, rank, and reason for action.

  3. 3. Track interventions

    Record human overrides, treatments, messages, and downstream actions.

  4. 4. Preserve exploration data

    Separate representative or randomized samples from policy-selected labels.

  5. 5. Evaluate long-term effects

    Measure diversity, opportunity, recurrence, and delayed outcomes.

  6. 6. Set loop safeguards

    Use caps, audits, fallback, and review for self-reinforcing behavior.

Example

Where naive retraining can reinforce the wrong pattern

These loops appear across product categories, and in several of them the size of the effect has been measured rather than assumed.

  • Search: high-ranked results receive the clicks that justify future high ranking — in the 2005 eye-tracking study the top result took 19 of 20 clicks when it was the more relevant of the top two, and the second took only 2 of 7 when it was.
  • Content moderation: only flagged content receives detailed labels, leaving false negatives poorly measured.
  • Healthcare: treatment changes the outcome a risk model predicted, and the better the treatment works the faster the model appears to degrade (JAMIA, 2019).
  • Credit: approval policy determines whose repayment becomes observable.
  • Predictive maintenance: successful interventions remove failures from future labels, and refitting on those labels can push successive scores toward predicting their own effect (AISTATS, 2021).
  • Education: recommendations shape which material students practice and therefore what mastery data exists.

Evaluate the policy, not only the static predictor

Once predictions change exposure or intervention, offline accuracy on historical labels becomes incomplete. The deployed unit is a policy. It is model scores plus thresholds, ranking, capacity, user interface, and human action.

Epic's proprietary sepsis prediction model makes the distinction concrete. Wong and colleagues validated it externally on 38,455 hospitalizations of 27,697 patients and published the result in JAMA Internal Medicine in June 2021. The hospitalization-level area under the curve was 0.63, with a 95% confidence interval of 0.62–0.64. The conclusion of the abstract is blunt: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.”

But the numbers a hospital actually lives with belong to the policy, not the predictor. At an alert threshold of 6, within the developer's recommended range, the system fired on 6,971 of the 38,455 hospitalizations — 18% of admissions. Of the 2,552 hospitalizations with sepsis, it flagged 183 (7%) in which the patient had not received timely antibiotics — the cases where an alert could add something a clinician had not already done. Move the threshold and both of those numbers move. The trained model is unchanged.

The same predictor also behaves differently where it is installed. A 2024 validation across two county emergency departments and 145,885 encounters measured a sensitivity of 14.7% and a positive predictive value of 7.6% in a 6-hour window. One model, two deployments, measured with different yardsticks, and neither found it strong.

A policy can create value with an imperfect predictor or cause harm with a high-scoring one, so evaluation should include counterfactual uncertainty, long-term outcomes, and who receives opportunities or burdens.

The same group that named the selective labels problem put the policy question in the same frame in February 2018. They worked from bail decisions in New York City. Cases there are assigned to judges quasi-randomly, which is how they got around the outcomes they could not observe. They report that “one policy simulation shows crime reductions up to 24.7% with no change in jailing rates, or jailing rate reductions up to 41.9% with no increase in crime rates”. Neither figure is a model accuracy. Both describe a decision rule: a score, a threshold, and who consequently goes home. That is the object the paper compares, and the object a deployment actually is.

Key takeaways