Feedback, Labeling, and Human Review Loops

A system that learns only from the cases its own policy sent people to look at will keep confirming that policy. That is proved, not supposed. In 2018 Ensign and colleagues built a mathematical model of predictive policing and proved why discovered-incident data — the incidents found because officers were sent to look — drives a runaway feedback loop. The loop sends police back to the same neighbourhoods regardless of the true crime rate. Incidents reported by residents rather than discovered by patrol reduce the loop. They do not remove it.

The output had already been measured. Kristian Lum and William Isaac ran PredPol's algorithm on Oakland drug-crime records, every grid square for every day of 2011, and set the result against public-health estimates of roughly equivalent drug use across racial groups. They published it in Significance in October 2016: “Black people would be targeted by predictive policing at roughly twice the rate of whites. Individuals classified as a race other than white or black would receive targeted policing at a rate 1.5 times that of whites.” The system was not measuring crime. It was measuring where it had already sent people, and then sending them there again.

A review queue works the same way. A content classifier routes uncertain examples to moderators and learns from their decisions. It never shows a moderator a confident mistake. So the next model gets better around the old boundary and keeps its blind spots everywhere else. Human feedback is collected through a policy, and that policy decides which labels exist, which reviewers see them, and which errors can be discovered at all.

The identity of the labeller travels further than anyone would want. Geva and colleagues looked at three NLP datasets in 2019. They “show that model performance improves when training with annotator identifiers as features, and that models are able to recognize the most productive annotators”. They also “show that often models do not generalize well to examples from annotators that did not contribute to the training set”. Their recommendation is structural: “test set annotators should be disjoint from training set annotators”. Part of what the model had learned was who wrote the labels.

Higher effort can provide stronger evidence, but no source is universally authoritative, and each rung is produced by a different collection policy.

Implicit interaction — clicks, dwell, completion, abandonment, workflow actions — is cheap, and the interface generates it. Explicit user feedback — ratings, flags, corrections, free-text complaints — states a meaning, but is selected by whoever bothered to send it. An operational outcome — resolution, repayment, maintenance result, downstream consequence — is objective, late, and shaped by the action that was taken. Expert annotation — a rubric-based label, an explanation, a structured adjudication — is the most expensive, and still argues with itself.

Only the top rung, independent audit and appeal, is sampled outside the ordinary selection process. That is the whole of its value. It is the only rung that can surface an error the system was confident about, because it is the only one the system did not choose.

Operators review model-flagged transactions under a strict time target. The score and the recommendation sit at the top of the screen. The reviewer's own scorecard rewards speed and agreement.

That queue does not merely produce weak labels. It fails a test two data-protection authorities have written down. A controller cannot escape Article 22 GDPR by fabricating human involvement. The Article 29 Working Party's guidance on automated decision-making and profiling says what the involvement has to be: “To qualify as human involvement, the controller must ensure that any oversight of the decision is meaningful, rather than just a token gesture. It should be carried out by someone who has the authority and competence to change the decision.” The UK Information Commissioner's Office repeats the test in its guidance on AI and data protection. A decision does not fall outside Article 22 just because a human has “rubber-stamped” it. Reviewers need the “authority and competence” to go against the recommendation. And the ICO names automation bias — “automation-induced complacency” — as a risk factor that can push a decision-support system into the scope of Article 22.

A user thumbs-down may indicate factual error, unwanted style, disagreement, or frustration with the interface. An expert label follows a rubric but may still be ambiguous. An operational outcome can be objective yet delayed and influenced by the action taken. These are not three grades of one signal. They answer different questions, and one of them can be moved a long way without moving the others at all.

The gap has a number on it. In 2024 Wen and colleagues gave time-constrained human subjects 3–10 minutes to judge model outputs against gold labels. After a standard RLHF pipeline the models convinced the subjects more without completing the tasks better: “RLHF also makes the model harder to evaluate: our subjects' false positive rate increases by 24.1% on QuALITY and 18.3% on APPS.” Approval moved by 24.1% on the reading-comprehension task and 18.3% on the programming task. Task performance did not follow it. Sharma and colleagues at Anthropic found the same pull from the other end. Across five state-of-the-art assistants and existing human preference data, a response matching a user's views is more likely to be preferred, and both humans and preference models prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time.

Store source, rubric, context, timestamp, reviewer role, confidence, and adjudication. Do not merge heterogeneous feedback into one generic target without modeling its meaning. An optimiser handed the merged column will find the cheapest signal in it, and the cheapest signal is approval.

Combining them can contaminate labels and weaken user recourse. Operational review supports a live decision under time and capacity constraints; it may use model context, is optimized for safe action, produces biased labels, and needs escalation and override. Training annotation creates evidence for learning or evaluation; it uses a stable rubric, can blind model output, supports redundancy and adjudication, and should sample beyond production flags. Appeal or audit reconsiders contested or systematically sampled decisions; it is independent from the original workflow, supports recourse, and should feed governance as well as labels.

Michigan ran the first lane without the other two. MiDAS auto-adjudicated unemployment fraud with no human review, and two state bodies later measured the result. The Michigan Supreme Court, deciding Bauserman v Unemployment Insurance Agency in 2022, recorded a study the agency had run on itself: “In addition, a study conducted by the Agency concluded that, during this same period, approximately 93% of the automated system’s fraud determinations were incorrect.”

The appeal lane had already registered the load. A February 2016 performance audit by the Michigan Office of the Auditor General found outstanding unemployment appeals rising from 2,280 as of June 2013 to 22,473 as of June 2015. Of the 51,207 appeals UIA sent to the Michigan Administrative Hearing System between October 2013 and June 2015, MAHS affirmed the agency in 8% of secondary misrepresentation cases and 21% of other cases. The independent lane was not a garnish on the operational one. It reversed roughly four decisions in five. It was the only lane telling the agency what its own labels were worth.

The loop should discover blind spots rather than merely confirm current behavior. Define feedback semantics, stating what each source can mean and which context is retained. Diversify sampling across uncertainty, random audit, rare slices, complaints, and missed outcomes. Protect label quality with rubrics, blinding, redundancy, adjudication, and reviewer calibration. Measure workflow burden — queue age, capacity, escalation, overrides, reviewer harm. Separate uses, deciding whether a piece of feedback changes a live action, a training set, an evaluation, or a governance record.

Step four is the one that gets skipped. The Oversight Board's policy advisory opinion on Meta's cross-check program, published in December 2022, shows what skipping it costs. Enrolment in cross-check exceeded the capacity to review it. “Meta told the Board, that, on average, it can take more than five days to reach a decision on content from users on its cross-check lists.” Mean time to a final decision in Early Response Secondary Review ran to about 12 days on average for content posted in the United States and about 17 days for Afghanistan and Syria. The longest a single piece of content sat in the ERSR queue was 222 days. The overturn rate Meta reported for ERSR ranged from 30% to 90% across time periods — a spread that is itself a measurement failure.

The number the Board could not obtain is the one this lesson is about. Meta did not provide the Board with information showing that it tracks whether cross-check decisions are more or less accurate than its normal quality control. The largest human review queue ever built could not say whether the humans made it better. The opinion carried 32 recommendations. Meta's own transparency record commits it not to operate these programs at a backlog, and to adhere to service-level agreements for review decisions.

Changing the queue, interface, reviewer incentives, or appeal process changes the data-generating process. Version these workflow components and analyze them when model behavior shifts.

Active learning is the cleanest demonstration, because there the acquisition policy is welded to the model that ran it. Lowell and colleagues reported in 2019: “We find that subsequently training a successor model with an actively-acquired dataset does not consistently outperform training on i.i.d. sampled data.” The benefits of current active-learning approaches, they find, do not generalise reliably across models and tasks. A 2022 replication by Munjal and colleagues reached the same place independently. Under identical experimental settings, uncertainty-, diversity- and committee-based active learning produce an inconsistent gain over the random sampling baseline, and marginal or no advantage under strong regularisation. A dataset chosen by last quarter's model is, in part, a fact about last quarter's model. The successor does not inherit the benefit.

The system should retain evidence that it can be wrong outside its own selection boundary.