Labels, Rewards, Delays, and Proxy Outcomes

An apparel recommender optimized purchase probability. It promoted visually attractive but poorly fitting items, increasing orders and also increasing returns, support contacts, and customer frustration several weeks later.

How many weeks is not a matter of taste, and no training window gets to decide it. In the EU the clock is written into statute. Article 9(1) of the 2011 consumer rights directive gives the buyer 14 days: “Save where the exceptions provided for in Article 16 apply, the consumer shall have a period of 14 days to withdraw from a distance or off-premises contract, without giving any reason, and without incurring any costs other than those provided for in Article 13(2) and Article 14.” Article 14(1) then grants a further 14 days to send the goods back, counted from the day that decision is communicated. The UK's implementing regulations set the same 14 days, running from the moment the goods come into the consumer's physical possession.

Add the two periods. A lawful return can arrive roughly four weeks after delivery — more than four weeks after the order, once shipping time is counted. So a thirty-day observation window on an apparel purchase label is a floor, not a round number. And every order placed inside the last month is not evidence of a good recommendation. It is an unresolved case, and the training set is about to record it as a success.

Label maturation ignored and reward leakage are the same mistake from opposite directions in time. The first marks an example negative before the outcome had a chance to arrive. The second lets information that arrived later into the target.

The first of the two has an industrial name and a measured price. Twitter's ads click-through system is trained continuously on a stream in which positive labels arrive late. In 2019 the team running it stated the failure plainly: “Naive strategies which consider any data point a negative example until a positive label becomes available tend to underestimate CTR, resulting in inferior user experience and suboptimal performance for advertisers.” They call those premature negatives fake negatives. Then they went after them. Five loss functions, compared on 668 million in-house examples. The result beat the previous state of the art by 3% relative cross entropy, and gained 55% in revenue per thousand requests over naive log loss in online experiments. Getting an honest holdback set at all meant waiting up to 9 hours past the evaluation date for an engagement label. The paper does not claim to know the right window length. That is worth remembering when a workflow step says simply "choose observation windows".

Recommendation labels often answer a narrower question than the product objective. A click label estimates interaction under a particular exposure policy. A purchase label ignores satisfaction. A watch-time label can favor length. A survey label may be sparse and non-random. Before a delayed outcome can be used at all, somebody has to write down four things: where the cutoff falls, how long the observation window runs, what happens to cases still open when it closes, and which exposure gets the credit. Recent examples cannot be labeled negative simply because the outcome has not arrived.

Alibaba's Entire Space Multi-Task Model, published in 2018, treats the chain impression, click, conversion as one sequence rather than two disconnected tasks. Its argument joins sparsity to bias, and the sparsity is not a figure of speech: “Table 1 shows the statistics of our experimental datasets, where number of samples for CVR task is just 4% of that for CTR task.” The released Public Dataset holds 84M impressions, 3.4M clicks and 18k conversions. The full Product Dataset holds 8,950M impressions, 324M clicks and 1,774k conversions. Modelling the chain over the entire space lifted CVR-task AUC from 66.00 for the base model to 68.56. That is a gain of 2.56 points on the public data, and 2.18 points on the 8.9-billion-sample product data. And the authors say 0.1% AUC is remarkable in this setting. That is what the delayed end of the chain is worth.

The other route is to change the label itself. YouTube's production ranker deliberately does not rank by click probability. It predicts expected watch time, using weighted logistic regression. Google's 2016 paper on it gives the reason: “Ranking by click-through rate often promotes deceptive videos that the user does not complete (“clickbait”) whereas watch time better captures engagement”. The weighting carries the whole idea. Retrain the same 1024→512→256 network with positive and negative examples weighted equally, and watch-time-weighted per-user loss rises by 4.1%. That is against a best measured loss of 34.6% on next-day holdout data, and 41.6% with no hidden layers. A later Google ranking system for the same product, from 2019, takes the multiple-objective route instead: several competing targets optimised in one network. None of these teams treats a single fast label as the goal. All of them pay for the delay somewhere in the design.

Outcomes arrive at different speeds. Immediate behavior — click, open, add-to-cart, play, or save — lands in a second. Session value follows within the visit: depth, task completion, abandonment, repeated reformulation, or frustration. A delayed user outcome lands in a week or a month: a return, a completion, a renewal, a complaint, long-term retention. Under Article 9(1) plus Article 14(1), a return may lawfully take four weeks to appear. Provider and platform outcomes accumulate over quarters — seller value, catalog health, inventory, revenue, margin, creator opportunity. The guardrail and harm row lands whenever somebody reports it: spam reports, unsafe exposure, excessive use, returns, churn, regulatory risk. A reward function has to choose among these five without waiting for the slow ones. That is exactly why the choice has to be written down. Left unwritten, it is inherited from whichever event the logging pipeline happened to make cheapest.

All three answer the question of what the model should be rewarded for. They differ in what they let a team see afterwards. A composite label buries the tradeoff inside a weight: simple downstream, hard to diagnose when a regression arrives, useful mainly when the combined events share one interpretation. Multi-task prediction keeps each outcome visible and inherits the problem of labels that arrive late. It is the shape Alibaba chose for the impression, click, conversion chain. It buys outcome-specific diagnostics at the price of missing labels and task conflict. A policy reward scores consequences over time, supports exploration and sequential decisions, and needs causal discipline to stay honest.

The last column's weakness has been measured directly. Four reinforcement-learning environments, each with a deliberately misspecified reward function, and agent capability varied along four axes: model capacity, action-space resolution, observation-space noise and training time. Pan and colleagues published the result in 2022: “More capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward than less capable agents.” Worse for anyone relying on a launch review, the collapse does not creep in. They report phase transitions — capability thresholds at which behaviour shifts qualitatively and true reward drops sharply. A reward that has been safe through every previous model size is not thereby safe through the next one.

If the horizon is left unstated, every later choice about windows and attribution gets made by accident. That is why the specification states it first and sets the guardrail stops last. State the decision horizon. Choose observation windows that match each outcome to its delay and maturity period. Define attribution: name which exposures can receive credit, and how overlapping touches are handled. Preserve separate outcomes rather than collapsing incompatible objectives before diagnostics are possible. Then set guardrail stops — outcome changes that block launch even when the primary proxy improves.

Step three is the one teams treat as bookkeeping, and there is a public dataset built precisely to show that it is not. Criteo released its Attribution Modeling for Bidding dataset on 10 October 2017: 16.5M impressions, 45K conversions and 700 campaigns, sampled from 30 days of live traffic, 2.4Gb uncompressed. It stores two different flags rather than one. A `conversion` flag is “1 if there was a conversion in the 30 days after the impression (independently of whether this impression was last click or not)”. An `attribution` flag is “1 if the conversion was attributed to Criteo, 0 otherwise”. They are kept apart because of the default rule the accompanying paper names: “The industry standard for conversion attribution is to credit only the last click in the 30 days before the conversion”. "A conversion happened" and "we were credited for it" are two columns. A specification that does not say which one the label means has already decided by default.