How machines learn
Feedback Loops and Selective Labels
Understand how recommendations, ranking, review policies, and interventions alter future behavior and which labels become visible.
By the end you can
- Explain how model actions can change future inputs, outcomes, and labels
- Distinguish reinforcing, balancing, and measurement feedback loops
- Recognize selective-label and exposure bias created by decision policies
- Design exploration, logging, and evaluation practices that preserve learning opportunities
Visual
Equal looks, unequal clicks: the model helps create the behavior it later learns from
A recommendation system does not merely observe preference. It chooses exposure, which affects clicks, watch time, purchases, and future training data. That the click reflects position as well as preference is not an inference — it was measured with an eye-tracker.
Users looked at the abstracts ranked 1 and 2 with almost equal frequency. They clicked the first one far more often. That was 2005, across 34 recruited subjects, 29 of them with usable gaze data. Joachims and his co-authors found the result worth remarking on: “It is very interesting that users click substantially more often on the first than on the second link, while they view the corresponding abstract with almost equal frequency.”
A second phase tested whether relevance rescues the signal. It ran 22 recruited subjects, 16 usable: 6 “normal”, 5 “swapped”, 5 “reversed”. In the unmanipulated normal condition, counting only the sessions where a user clicked exactly one of the top two, the top result was clicked in 19 of the 20 cases in which it was the more relevant of the pair. Where the second result was the more relevant, it was clicked in only 2 of 7. Then a proxy secretly swapped the top two: “there is still a strong bias to click on link one even if the second abstract is more relevant.”
Microsoft Research reached the same conclusion three years later by a different route. In 2008 it reported that “the probability of click is influenced by a document's position in the results page”. Rank one is therefore partly a self-fulfilling prophecy. The ranking the model produced yesterday is inside the click it trains on tomorrow.
- 1
Model ranks items
Past data and objectives determine what is shown.
- 2
Users receive selective exposure
Unshown items cannot receive ordinary feedback.
- 3
Behavior is recorded
Clicks and dwell time reflect preference, position, interface, and availability.
- 4
Training data is updated
Observed interactions favor previously exposed items.
- 5
Next model reinforces patterns
Popularity and blind spots can grow unless the loop is managed.
Comparison
Feedback can amplify, counteract, or distort
The effect depends on how predictions change the process. Each of the three shapes has a worked instance later in this lesson.
A reinforcing loop sends patrols back to the neighbourhoods they already policed — PredPol on Oakland records. A balancing loop is the sepsis alert or the maintenance visit that removes the very event the score predicted. The score then appears to fail in proportion to how well the intervention works. A measurement loop is judicial bail: the outcome exists only for the defendants a judge released, so the metric describes the policy's coverage as much as the model's skill.
Reinforcing loop
Model actions increase the future prevalence of patterns they favor.
- Popularity can compound
- Selected groups receive more opportunity
- Errors can self-confirm
- Example: recommended creators gain more engagement
Balancing loop
Actions reduce the condition the model predicts.
- Successful intervention makes positives disappear
- Naive evaluation may punish helpful models
- Counterfactual outcomes are missing
- Example: maintenance prevents failure
Measurement loop
The policy changes which cases receive labels or attention.
- Reviewed cases are not representative
- Unselected outcomes stay unknown
- Metrics reflect policy coverage
- Example: only high-risk transactions are investigated
Case
PredPol on Oakland's records: twice the patrols, equal drug use
PredPol's published algorithm, run on Oakland police records of drug crimes, would have targeted black neighborhoods at roughly twice the rate of white ones. Estimates from public-health surveys put illicit drug use in Oakland at roughly equal rates across racial and income groups. Lum and Isaac ran that experiment in Significance in October 2016.
The mechanism was modelled two years later, by five researchers at the first FAT* conference. In a system updated on the crime its own patrols discover, “police are repeatedly sent back to the same neighborhoods regardless of the true crime rate”. Incidents reported by residents “can attenuate the degree of runaway feedback” but “cannot entirely remove it”. The records then measure where the patrols went, not where the crime was.
You observe outcomes for decisions that were made
A lender observes repayment for approved applicants, not for rejected ones. A hiring system observes job performance for hired candidates. A fraud team verifies cases sent to investigation.
The label dataset is therefore conditional on previous decisions. Training a model as if unobserved cases were ordinary negatives can reproduce the old policy and hide missed opportunities or harms.
Missing outcomes are often caused by decision policy, not random chance.
Case
Bail: an outcome recorded only for the defendants a judge released
The term is not informal. Five researchers named “the selective labels problem” at KDD in 2017, and judicial bail is the worked case. Whether a defendant fails to appear in court is observed only for the defendants some judge chose to release. For the detained there is no outcome at all.
Their proposed answer is contraction. It does not try to impute the missing outcomes. It compares model and judge “without resorting to counterfactual inference”. It harnesses instead the heterogeneity of the judges themselves. It leans on the most lenient of them, whose released set already contains many of the cases stricter judges detained.
Key idea
A good model can make its own predictions look wrong — and retraining makes it worse
If a model predicts machine failure and maintenance prevents it, the recorded outcome is “no failure.” Treating this as a false positive ignores the intervention that changed the result.
This is a named and published failure mode, not an intuition. A 2019 paper in the Journal of the American Medical Informatics Association is titled Prognostic models will be victims of their own success, unless… Its abstract states the paradox in a sentence: “The more effective a model and intervention(s) are at improving outcomes, the faster a model will appear to degrade.” The better the response to the alert, the worse the alert looks in the record.
Refitting on those post-intervention labels does not repair the distortion. It compounds it. A 2021 AISTATS paper formalises the existing score as a causal pathway into the outcome. Naive repeated updating, it shows, can drive successive scores toward a point where “they predict their own effect”, or into a stable oscillation between two values — and “even if model-fitting procedures improve, actual performance may worsen”.
Log the prediction, action, timing, and intervention. Evaluation may need to compare policies, use controlled experiments, or define outcomes that account for successful prevention.
Prediction quality and intervention effect cannot always be separated from observational labels.
Some randomization preserves evidence: Twitter's 1% who never got the algorithm
Small controlled exploration can expose items, inspect cases, or choose actions outside the current policy. This creates data about alternatives and reduces blind spots.
One platform kept such a holdout running for years. When Twitter introduced machine-learning personalisation of the Home timeline in 2016, it permanently excluded a randomly chosen 1% of global users. They still receive a reverse-chronological feed. The PNAS paper that used them, published on 4 January 2022, opens its abstract with the design: “We provide quantitative evidence from a long-running, massive-scale randomized experiment on the Twitter platform that committed a randomized control group including nearly 2M daily active accounts to a reverse-chronological content feed free of algorithmic personalization.”
What that group made measurable, the logs of the personalised feed could not have shown, because there is nothing in them to compare against. Using the holdout, the authors found that in 6 of the 7 countries studied the mainstream political right was algorithmically amplified more than the mainstream left. In the US media landscape, amplification favoured right-leaning news sources.
A preserved baseline also outlives the analysis its owner ran. An independent Berkeley team applied the same reverse-chronological comparison to 806 Twitter users between 11 and 27 February 2023. Relative to that baseline, engagement-based ranking amplified anger by 0.47 SD, partisanship by 0.24 SD and out-group animosity by 0.24 SD. Twitter built the control group for its own purposes. Somebody else measured with it.
Exploration carries cost and risk. High-stakes domains require ethical review, bounded options, informed consent where applicable, and safer designs such as expert adjudication, retrospective labeling, or staged pilots.
Steps
Design observability before deploying the policy — in the EU, it is Article 15(4)
Feedback loops become easier to study when the system records the choices that generated the data: the choice set, the policy context, the interventions, a protected slice of exploration data, long-term effects, and safeguards on self-reinforcing behaviour.
For one class of system the last two steps are not craft advice. They are law, with an article number. Regulation (EU) 2024/1689, the Artificial Intelligence Act, adopted 13 June 2024, states in Article 15(4): “High-risk AI systems that continue to learn after being placed on the market or put into service shall be developed in such a way as to eliminate or reduce as far as possible the risk of possibly biased outputs influencing input for future operations (feedback loops), and as to ensure that any such feedback loops are duly addressed with appropriate mitigation measures.”
The same obligation reaches back into the data pipeline. Recital 67 requires data-governance practices to attend to bias 'especially where data outputs influence inputs for future operations (feedback loops)'. For a continuously-learning high-risk system in the EU, "manage the loop" is a documented duty attached to a numbered provision. It is not a maturity level a team can defer.
1. Log the choice set
Record which options were available, shown, hidden, or infeasible.
2. Log policy context
Store model version, scores, threshold, rank, and reason for action.
3. Track interventions
Record human overrides, treatments, messages, and downstream actions.
4. Preserve exploration data
Separate representative or randomized samples from policy-selected labels.
5. Evaluate long-term effects
Measure diversity, opportunity, recurrence, and delayed outcomes.
6. Set loop safeguards
Use caps, audits, fallback, and review for self-reinforcing behavior.
Example
Where naive retraining can reinforce the wrong pattern
These loops appear across product categories, and in several of them the size of the effect has been measured rather than assumed.
- Search: high-ranked results receive the clicks that justify future high ranking — in the 2005 eye-tracking study the top result took 19 of 20 clicks when it was the more relevant of the top two, and the second took only 2 of 7 when it was.
- Content moderation: only flagged content receives detailed labels, leaving false negatives poorly measured.
- Healthcare: treatment changes the outcome a risk model predicted, and the better the treatment works the faster the model appears to degrade (JAMIA, 2019).
- Credit: approval policy determines whose repayment becomes observable.
- Predictive maintenance: successful interventions remove failures from future labels, and refitting on those labels can push successive scores toward predicting their own effect (AISTATS, 2021).
- Education: recommendations shape which material students practice and therefore what mastery data exists.
Evaluate the policy, not only the static predictor
Once predictions change exposure or intervention, offline accuracy on historical labels becomes incomplete. The deployed unit is a policy. It is model scores plus thresholds, ranking, capacity, user interface, and human action.
Epic's proprietary sepsis prediction model makes the distinction concrete. Wong and colleagues validated it externally on 38,455 hospitalizations of 27,697 patients and published the result in JAMA Internal Medicine in June 2021. The hospitalization-level area under the curve was 0.63, with a 95% confidence interval of 0.62–0.64. The conclusion of the abstract is blunt: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.”
But the numbers a hospital actually lives with belong to the policy, not the predictor. At the vendor-recommended alert threshold of 6, the system fired on 6,971 of the 38,455 hospitalizations — 18% of admissions. It identified 183 of the 2,552 sepsis patients (7%) who had not received timely antibiotics. Move the threshold and both of those numbers move. The trained model is unchanged.
The same predictor also behaves differently where it is installed. A 2024 validation across two county emergency departments and 145,885 encounters measured a sensitivity of 14.7% and a positive predictive value of 7.6% in a 6-hour window. One model, two deployments, two performances.
A policy can create value with an imperfect predictor or cause harm with a high-scoring one, so evaluation should include counterfactual uncertainty, long-term outcomes, and who receives opportunities or burdens.
The same group that named the selective labels problem put the policy question in the same frame in February 2018. They worked from bail decisions in New York City. Cases there are assigned to judges quasi-randomly, which is how they got around the outcomes they could not observe. They report that “one policy simulation shows crime reductions up to 24.7% with no change in jailing rates, or jailing rate reductions up to 41.9% with no increase in crime rates”. Neither figure is a model accuracy. Both describe a decision rule: a score, a threshold, and who consequently goes home. That is the object the paper compares, and the object a deployment actually is.
Key takeaways
- Model outputs change future inputs, outcomes, exposure, and which labels become observable. In the 2005 eye-tracking study users viewed results 1 and 2 with almost equal frequency and clicked the first far more often, so the ranking is inside the click that trains the next ranking.
- Reinforcing loops amplify favored patterns — PredPol on Oakland drug records would have targeted black neighborhoods at roughly twice the rate of white ones. Balancing loops suppress the predicted event. Measurement loops shape which evidence exists at all.
- Selective labels arise when outcomes exist only for cases approved, hired, treated, investigated, or otherwise chosen — bail failure-to-appear is recorded only for defendants some judge released.
- Interventions invalidate naive error labels, in proportion to how well the intervention works: the more effective the model and its intervention, the faster the model appears to degrade. Naive refitting can drive scores toward predicting their own effect.
- Bounded exploration preserves learning about alternatives. Twitter's permanently excluded 1% of global users — nearly 2M daily active accounts on a reverse-chronological feed — made algorithmic amplification measurable in 6 of 7 countries. It still requires explicit ethical and operational safeguards.
- After deployment, evaluate the complete policy: at Epic's recommended sepsis threshold of 6, alerts fired on 18% of admissions to identify 7% of the sepsis patients who had missed timely antibiotics, and in the EU Article 15(4) of Regulation (EU) 2024/1689 makes managing that loop a legal duty.