Recommender systems
Product Surfaces, Objectives, and Non-Goals
Define surface-specific objectives, constraints, and non-goals for home feeds, search, notifications, checkout, and other recommendation contexts.
By the end you can
- Define recommendation objectives in the context of a specific product surface
- Distinguish item, slate, row, message, and sequence-level decisions
- Identify non-goals and guardrails that prevent proxy optimization
- Assess whether model reuse across surfaces is justified by a shared decision contract
Key idea
Facebook cut 50 million hours a day on purpose
“Engagement” is not a portable objective. Its meaning changes with the action, the surface, the time horizon, and the cost imposed on the user.
The strongest evidence that engagement is a choice rather than a fact is a company lowering its own headline engagement number on purpose. Facebook did that. Video-recommendation and other quality changes in the fourth quarter of 2017 cut time spent on the site by roughly 5% that quarter. On the earnings call of 31 January 2018, Mark Zuckerberg put the cut in hours: “we made changes that reduced time spent on Facebook by an estimated 50 million hours every day to make sure that people's time is well spent”.
Then he said what the metric was worth. “So let me be clear: helping people connect is more important than maximizing the time they spend on Facebook.” Time spent by itself, he added, had never been the most important driver of the business. The ranking change behind the drop had been announced on 11 January 2018 by Adam Mosseri, then head of News Feed, in a post called “Bringing People Closer Together”.
A metric a company can decide to move downward, and be right to, is not a definition of success. It does not travel between surfaces on its own.
Two surfaces reporting the same engagement number may be counting two different user experiences, and the number will not say which.
Example
Amazon.com gave the shopping cart its own recommender
Amazon's shopping cart has its own recommender. It is not the homepage recommender with a different template around it.
That was already true in 2003, when Amazon described the production system in print. The homepage carries “Your Recommendations”. The cart runs a separate feature that recommends from the items currently in the cart, and its rationale belongs to the moment: “The feature is similar to the impulse items in a supermarket checkout line, but our impulse items are targeted to each customer.”
One catalog and one customer, two different questions. That is the pattern to hold against any proposal to reuse a ranker. The same user arrives at each surface having done something different to get there, and owing the system something different once there.
- Homepage: Broad discovery and orientation matter before the user has expressed a narrow intent. That is the surface Amazon gives to “Your Recommendations”.
- Product page: Complementarity and substitution matter relative to the item being viewed. The homepage has no item to anchor that comparison.
- Cart: Compatibility, shipping, price, and decision fatigue constrain what is worth adding. Amazon answered the cart with targeted impulse items, not with the homepage's ranked list.
- Notification: Whether and when to interrupt may matter more than which item scores highest. Pinterest built a system that sets notification volume per user, aimed at long-term engagement, and reported in 2018 what happened: “The new system has been deployed to production at Pinterest in mid 2017 and significantly reduced notification volume and improved CTR of notifications and site engagement metrics compared with the previous machine learning approach.” The win came from sending less.
- Post-purchase: Support, replenishment, and delayed needs differ from immediate conversion goals. A ranker tuned to close a sale is answering a question the user has already finished asking.
At Netflix, search gets its own algorithms
The same person can have different intents within minutes. A homepage may support exploration. A search box expresses a precise task. A notification has to justify interrupting someone.
Netflix put that split in print in 2015, with the shares attached. Gomez-Uribe and Hunt wrote: “Our recommender system is used on most screens of the Netflix product beyond the homepage, and in total influences choice for about 80% of hours streamed at Netflix. The remaining 20% comes from search, which requires its own set of algorithms.”
Search is not the homepage ranker with a query bolted on. It shares the catalog and the subscriber with every other screen. It is still built separately, because what the user is doing there is different.
The same paper puts a figure on getting the whole arrangement right: “We think the combined effect of personalization and recommendations save us more than $1B per year.” The saving is attributed to reduced monthly churn — a delayed outcome, not a click.
So the owner of a surface should settle six things before anyone proposes reusing a model on it: the decision unit, the candidate universe, the horizon, the capacity, how much repetition is acceptable, and what the user can control.
Reusing a model on a new surface without resettling those six choices ships one surface's assumptions onto another.
Case
YouTube chose watch time over clicks; Alibaba found the sample was wrong
A surface rarely has one objective. Which objective wins is a decision somebody makes and writes down.
YouTube's ranker aims at watch time, not clicks. Google's 2016 paper says why: “Our final ranking objective is constantly being tuned based on live A/B testing results but is generally a simple function of expected watch time per impression. Ranking by click-through rate often promotes deceptive videos that the user does not complete (“clickbait”) whereas watch time better captures engagement”. Two candidate objectives on one surface, both of them measurable. The team rejected the easier one because of what optimizing it does to the catalog.
Google's 2019 account of the YouTube next-watch ranker names two problems at once: “the presence of multiple competing ranking objectives, as well as implicit selection biases in user feedback”. Its answer is “soft-parameter sharing techniques such as Multi-gate Mixture-of-Experts”.
Alibaba's ESMM paper, from 2018, is the second half of that. It models the “sequential pattern of user actions, i.e., impression -> click -> conversion”, on data “gathered from Taobao’s recommender system”. The abstract says what it is fixing: “conventional CVR models are trained with samples of clicked impressions while utilized to make inference on the entire space with samples of all impressions. This causes a sample selection bias problem.”
Read that slowly. A conversion model trained only on clicked impressions is asked at serving time to score every impression. The population it was fitted on is not the population it scores. That gap is a selection bias, not a tuning failure. Schnabel and colleagues had formalised it in 2016: “Most data for evaluating and training recommender systems is subject to selection biases, either through self-selection by the users or through the actions of the recommendation system itself”. No amount of tuning on the clicked subset closes it.
Visual
A surface contract, with the middle rows written as numbers
A surface contract is written from the top down. It opens on what is known about the user right now — intent, urgency, expertise, session context. It closes on the non-goals: the metrics this surface refuses to chase even when they move.
The two middle rows are the ones teams leave abstract, and both can be written as numbers. Start with the decision unit and its capacity. Netflix states its own outright: “There are typically about 40 rows on each homepage (depending on the capabilities of the device), and up to 75 videos per row; these numbers vary somewhat across devices because of hardware and user experience considerations.” About 40 rows of up to 75 videos is a different decision from picking one item. A contract that says only “items” has not yet been written.
For large platforms, the constraints row and the user-control line are no longer a matter of design taste. The Digital Services Act was adopted on 19 October 2022. Article 38 reads: “In addition to the requirements set out in Article 27, providers of very large online platforms and of very large online search engines that use recommender systems shall provide at least one option for each of their recommender systems which is not based on profiling as defined in Article 4, point (4), of Regulation (EU) 2016/679.” Article 27 already carries the duty to disclose the main ranking parameters. On those services, what the user can control is a numbered obligation. It belongs in the contract as one.
Between those rows sits the outcome horizon — which immediate and delayed outcomes count, and when they become observable. Beneath everything, the non-goals.
User state
What is known about intent, urgency, expertise, and session context?
Decision unit
Is the system choosing one item, a row, a page, a message, or a sequence?
Outcome horizon
Which immediate and delayed outcomes count, and when become they observable?
Constraints
Which inventory, legal, safety, frequency, and capacity rules apply?
Non-goals
Which tempting metrics must not define success on this surface?
Steps
Write a surface-specific recommendation brief
Five entries make a brief that another team can argue with rather than merely read. They run from the moment being served to the outcomes the surface refuses to chase. Each has a published example to imitate.
1. Describe the moment. State what brought the user to the surface and which intent is observable. An Amazon shopping cart holds an item set the homepage does not. That is why the cart got a feature of its own.
2. Fix the choice unit. Item, slate, row, message, or sequence. Then give its capacity a number, the way Netflix gives about 40 rows of up to 75 videos per row.
3. Name desired outcomes. Separate immediate behavior, delayed value, and guardrails. YouTube choosing expected watch time per impression over click-through rate is this step performed in public.
4. Record constraints. Eligibility, repetition, timing, and policy. The Article 38 duty to offer at least one recommender option not based on profiling is a constraint of exactly this kind.
5. Declare non-goals. List the metrics the team will monitor but refuse to optimize directly. Pinterest's deployed system improved notification CTR and site engagement while reducing notification volume. A brief that optimized volume would never have shipped it.
1. Describe the moment
State what brought the user to the surface and which intent is observable.
2. Fix the choice unit
Define item, slate, row, message, or sequence and its capacity.
3. Name desired outcomes
Separate immediate behavior, delayed value, and guardrails.
4. Record constraints
Include eligibility, repetition, timing, and policy requirements.
5. Declare non-goals
List metrics the team will monitor but refuse to optimize directly.
Example
Horizon collapse, measured: a 50% ad-load cut at Google
Objective transplant is the mistake teams make on purpose, believing they are reusing work. No explicit non-goals is the one they make by omission. It is also the reason the transplanted metric is never challenged once it starts moving.
Horizon collapse is the one that has been measured. In 2015 Google reported that measuring long-term user learning licensed “a 50% reduction of the ad load on Google’s mobile search interface”. The paper's introduction describes what that cost: “Reducing the mobile ad load strongly improved the user experience but was a substantially short-term revenue negative change; with our work, the long-term revenue impact was shown to be neutral.”
One lever, two horizons, opposite signs. A team reading only the short-term number would have reversed a decision that cost it nothing.
- Objective transplant: A metric that works on one surface is copied to another without checking user intent, as though the catalog were the contract.
- Unit confusion: Teams evaluate item scores while the product displays rows or pages. A Netflix homepage of about 40 rows of up to 75 videos is judged as a page, not as its best single tile.
- Horizon collapse: Immediate clicks displace slower outcomes such as retention, learning, or trust. The Google mobile ad-load cut was substantially short-term revenue negative and long-term revenue neutral.
- Constraint afterthought: Eligibility and frequency caps get added after ranking, and results turn unstable. Obligations like the Article 38 non-profiling option are part of the surface's definition, not a filter bolted on at the end.
- No explicit non-goals: The team optimizes any metric that moves, even when it rewards interruption or repetition. That is the opposite of the discipline behind a roughly 5% cut in time spent on Facebook in the fourth quarter of 2017.
Key idea
The reuse test
Reuse a model only when the surfaces share the same decision contract, not merely the same catalog and user table. Amazon's homepage and shopping cart shared both, and still got different features for different moments. Netflix's homepage and search share both, and search still “requires its own set of algorithms”.
Ask what the new surface asks the user to do before asking whether the old model can score it.
Key takeaways
- A model is reusable only after the product decision has been shown to be reusable. Netflix's search shares the catalog and the subscriber with the homepage, and still “requires its own set of algorithms”.
- The same person can have different intents within minutes, and the split shows up in the numbers: about 80% of hours streamed at Netflix influenced by recommendations, the remaining 20% coming from search.
- “Engagement” is not a portable objective. Its meaning changes with the action, surface, time horizon, and cost imposed on the user — which is why Facebook could ship changes that removed an estimated 50 million hours a day of time spent and call them right.
- User state opens the surface contract: intent, urgency, expertise, session context. It settles the decision unit and its capacity, which should be written as a number — about 40 rows on a Netflix homepage, up to 75 videos per row.
- Objective transplant remains a practical risk: a metric that works on one surface is copied to another without checking user intent. Click-through rate would have travelled onto YouTube ranking had Google not chosen expected watch time per impression instead.
- Some metrics belong on the dashboard without belonging in the objective. Declaring those non-goals keeps a surface from quietly optimizing them — Pinterest improved notification CTR and site engagement by significantly reducing notification volume.