Learning Approaches: Start With the Feedback

A fraud filter, a warehouse robot, and a customer-segmentation study may all use machine learning, and they do not learn from the same kind of evidence. Two of the three have been written down in public with their figures attached. The figures are where the distinction stops being a slogan.

Fraud prediction. Frauds are about 0.2% of transactions in the real card data behind one 2018 study, and the supervision arrives in two streams that behave nothing alike. The authors worked with their industrial partner to set down how card-fraud labels actually reach a model. “Overall, there are two types of supervised information: 1) feedbacks provided by investigators that are limited in number but refer to recent transactions and 2) delayed supervised transactions, which are the vast majority for which the labels become available after several days (e.g., one month).” — Dal Pozzolo and colleagues. Only alerted transactions are checked promptly, a handful of investigator verifications a day. Everything else waits out what their abstract names as “verification latency (only a small set of transactions are timely checked by investigators)”, roughly a month, after which undisputed transactions are assumed genuine by default. So “delayed and selective labels” has a shape. The delay is about a month, the selection is the alert queue, and the default answer for an unexamined transaction is genuine.

Warehouse control. A bin-picking robot learned to grasp from 580,000 real-world grasp attempts, collected autonomously across 7 robots, about 800 robot hours. The teaching signal is one bit. “The reinforcement learning algorithm receives a binary reward for lifting an object successfully, and no other reward shaping.” That is QT-Opt, trained in 2018 by Kalashnikov and ten co-authors. The policy reached 96% grasp success on previously unseen test objects. The accompanying Google Research blog post reports that “the QT-Opt approach succeeded in 96% of the grasp attempts across 700 trial grasps on previously unseen objects”, and that “Compared to our previous supervised-learning based grasping approach, which had a 78% success rate, our method reduced the error rate by more than a factor of five.” Table 1 of the paper sets the two beside each other: 96% against 78%, on the same hardware and the same task. What changed was the feedback contract.

Customer exploration. Here no outcome arrives at all. Analysts hold behaviour records and no agreed customer categories, so the result is groups or representations, and it is validated by interpretation rather than measured against an answer key.

A learning approach is a contract between available experience and the behavior you want. The same model family can participate in different approaches, and different model families can solve the same supervised task.

Begin with the signal, not the fashionable algorithm. Ask who or what provides correction, how expensive that signal is, and whether it matches the outcome that matters. The QT-Opt result is the cleanest version of that argument: one robot fleet, one bin-picking task, 96% under a binary lift reward against 78% under supervised grasp prediction. Nobody bought that gap with a better architecture.

Learning signals vary. They differ in how directly they specify the desired output, and more indirect feedback often expands scale while introducing ambiguity and additional assumptions. Direct labels sit at one end: each example carries an intended answer, a category or a numeric target. Unlabeled structure sits at the other: the learner extracts patterns from observations with no task-specific answer anywhere. Between them are three rows with published price tags, and the prices are the point.

Comparisons and demonstrations. One 2017 experiment threw away the reward function entirely and put non-expert people in its place, choosing between pairs of trajectory segments. “We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than 1% of our agent’s interactions with the environment.” — Christiano and five co-authors, at NIPS 2017. The showpiece is a Hopper backflip, a behaviour nobody wants to write a reward function for. It “was trained using 900 queries in less than an hour”. DeepMind’s accompanying post puts it at around 900 pieces of feedback, and adds that “as little as 30 minutes of feedback from a non-expert is enough to train our system”. That is the row’s real content. Not that people can show what they want without specifying it, but that 900 pairwise comparisons and under an hour was the whole bill.

Rewards and outcomes. The system receives consequences after acting, sometimes long after. QT-Opt’s single lift bit, paid for with 580,000 grasps, is what that costs when the environment is physical.

Weak or partial supervision. Rules, distant databases, clicks and incomplete annotations give noisy guidance, and Snorkel measured what the noise costs. It trained models with no hand-labelled training data at all, using instead user-written labelling functions whose accuracies and correlations are unknown. From the 2017 abstract: “In two collaborations, with the U.S. Department of Veterans Affairs and the U.S. Food and Drug Administration, and on four open-source text and image data sets representative of other deployments, Snorkel provides 132% average improvements to predictive performance over prior heuristic approaches and comes within an average 3.60% of the predictive performance of large hand-curated training sets.” In the user study, subject-matter experts built models 2.8x faster than seven hours of hand labelling. Weak supervision is not free accuracy. It is 3.60% on average given up to avoid the label bill.

Consider a support platform. It wants to reduce time to resolution, and that product goal alone does not select one learning approach.

The cheapest route on that list is also the one that lies to you most politely. Clicks went under an eye-tracker at SIGIR in 2005, and for some participants the top two results were secretly swapped: 34 people in Phase I, usable eye data for 29, and 22 in Phase II, usable for 16, split across normal, swapped and reversed conditions. “First, we show that there is a “trust bias” which leads to more clicks on links ranked highly by Google, even if those abstracts are less relevant than other abstracts the user viewed.” — Joachims and four colleagues. Users kept clicking the first link even when the second abstract was the more relevant one, which is why the paper concludes that “we conclude that clicks are informative but biased”. Microsoft researchers reproduced the effect at web scale in 2008 by perturbing the rankings of a major search engine: “A key source of bias is presentation order: the probability of click is influenced by a document's position in the results page”. A ranking model trained on raw clicks is therefore learning, in part, where things were shown.

Use these questions first. They come before any discussion of algorithms or architecture. Name the decision — what output or action must the system produce, and when. Inventory the feedback: labels, outcomes, demonstrations, preferences, rules, unlabeled records. Measure the delay before reliable feedback appears, and note who can observe it. Test alignment: does the available signal reward the behavior the product actually values. Price the signal, counting annotation cost, interaction risk, expert time and privacy constraints. And preserve the alternatives — rules, retrieval, process redesign, or no automation.

Steps three and four are the ones teams skip, and Google Flu Trends is what skipping them looks like. The proxy was search queries, fast and cheap. The outcome was flu prevalence, slow and expensive. The fast signal correlated well enough to look like a win, and then it drifted: “GFT also missed by a very large margin in the 2011–2012 flu season and has missed high for 100 out of 108 weeks starting with August 2011”. Worse for the design case, three-week-old CDC data projected current prevalence better than GFT did. The slow ground truth, stale, still beat the fresh proxy. The diagnosis, from Lazer and three co-authors in Science in 2014, is one sentence: “In short, the initial version of GFT was part flu detector, part winter detector.” A signal that arrives quickly and a signal that measures the thing you care about are two different questions. 100 weeks in 108 is what it costs to answer only the first.

This path compares the major ways systems learn: direct labels, hidden structure, consequences, demonstrations, weak signals, transfer, and ongoing updates. It focuses on choosing and combining approaches responsibly.

Later paths examine algorithms, optimization, metrics, data engineering, mathematics, and production operations in depth; here, the goal is to recognize the learning problem before selecting machinery.