Multi-Objective Ranking and Constraint Handling

Spotify's most valuable listeners are its most varied ones. High consumption diversity is strongly associated with conversion and retention — the outcomes the business is run on. Spotify measured this across millions of users and published it in 2020.

The same paper found the other half. Of the mechanism that fills much of the listening, the abstract says: “However, we also find that algorithmically-driven listening through recommendations is associated with reduced consumption diversity.” The users who do become more diverse get there by shifting away from algorithmic consumption toward organic consumption.

Nothing in that arrangement announces itself as a tradeoff. Nobody set a weight to buy engagement with diversity. The quality that conversion and retention travel with simply thins out where recommendations do more of the choosing. That is what a multi-objective failure looks like from inside the team. Not a policy someone wrote, but an objective nobody put in the score, moving anyway.

Recommendation systems often predict several outcomes and combine them through weighted sums, constrained optimization, lexicographic rules, Pareto methods, or re-ranking. How they are combined determines which tradeoffs the platform is willing to make.

LinkedIn's feed has to balance four things that pull against each other: clicks, viral actions, job applications, and advertising revenue. In 2018 LinkedIn published how it does that. A composite utility function, a Gaussian process over the control parameters, Bayesian optimization tuning them online — real machinery, fully deployed. On the Feed itself the three metrics actually balanced are Engaged Feed Sessions, Viral Actions and Job Applies.

Then one sentence in the abstract says what the machinery is not for: “We assume that the desired balance of metrics is known from business considerations.”

The optimizer picks the parameters. People pick the balance the parameters are aimed at. A team that cannot say who made that choice has not removed the choice, only its author.

Hard requirements should remain constraints or gates. Soft objectives can trade only within approved ranges, with separate monitoring and owners.

Enumerate the stakeholders and outcomes before any weight is chosen: user, provider, platform and societal effects separated rather than summed. Assign change authority after every weight is chosen: versioned weights, limits, owners, rationale and rollback. Hard constraints, calibrated heads and tradeoff curves are the three steps in between.

Step two, classifying hard and soft requirements, is what stops non-compensable requirements from becoming tiny penalties. One published system shows the alternative in form rather than in intention. Its provider-side objective is not a term added to a score. Attention accumulated across a series of rankings is made proportional to accumulated relevance, and the quantity that gets bounded is ranking quality. The paper says so in one line: “We formulate the challenge of achieving amortized individual fairness subject to constraints on ranking quality as an online optimization problem and show that it can be solved as an integer linear program.” That is Biega and colleagues, in 2018.

Notice which side of the arrangement each objective is on. The fairness requirement is the thing being achieved. Ranking quality is the thing being constrained. Swap them and you have a penalty term, and a large enough relevance estimate can always outbid a penalty term.

Step three calibrates the objective heads. Each score goes on an interpretable, monitored scale, so a weight means what it claims. Step four tests the tradeoff curves — Pareto fronts, constraint feasibility across the candidate set, and how both vary by segment. That happens before users meet the fallback, not after. Step five hands the result to a named owner who can move it and reverse it.

The top layer is not an objective at all. Eligibility and safety are the things that may never be traded for relevance or revenue. Below that line the trading starts. User value — task success, satisfaction, retention, trust, welfare — competes openly with provider and ecosystem value: opportunity, supply health, creator or seller outcomes. It competes with platform value too: revenue, cost, capacity and strategic priorities. Then come the experience qualities — freshness, diversity, novelty, pacing and repetition. They decide how the trade feels to the person looking at it.

The provider layer is the easiest to underestimate, because the ranking never states what it is allocating. Take a job search. Three male applicants carry relevance 0.80, 0.79 and 0.78. Three female applicants carry 0.77, 0.76 and 0.75. Rank by predicted relevance, apply the standard position-bias discount 1/log(1+j), and “the female applicants will get 30% less exposure – even though the average difference in relevance between male and female applicants is just 0.03”. That is the opening example in Singh and Joachims, 2018.

No weight was chosen to produce that. There is no revenue objective in the example and no engagement head to blame. A difference of 0.03 in score becomes 30% of the opportunity, purely through the shape of the ranking and the way attention falls down a list. That is why provider value is a layer in the hierarchy rather than a courtesy. A ranking allocates it whether or not anyone put it in the objective set.

Scale dominance decides the tradeoff by accident: the head with the wider numeric range or the greater certainty wins whatever the weights say. Stakeholder invisibility decides it by omission, leaving affected users, providers or long-term effects out of the objective set entirely.

Weight drift is the quietest of the five, because the declared policy does not change at all. Modern networks are badly calibrated, and the effect tracks architecture. On CIFAR-100 a 5-layer LeNet with 44.9% error has an average confidence closely matching its accuracy. A 110-layer ResNet with 30.6% error is substantially overconfident. The more accurate model is the worse calibrated one. Guo and colleagues measured that in 2017, and had not expected it: “While neural networks today are undoubtedly more accurate than they were a decade ago, we discover with great surprise that modern neural networks are no longer well-calibrated.”

Put that model behind an unchanged weight vector and the effective policy has moved. A head that reports larger numbers for the same underlying event buys more of the score at the same coefficient. The upgrade nobody logged as a policy change was one.

The recommended fix is temperature scaling, a single-parameter variant of Platt scaling. It is cheap. There is no excuse for a monitored head drifting unmeasured.

Report each objective before and after combination, constraint violation, candidate-set feasibility, segment outcomes, and sensitivity to weight changes. Online experiments need guardrails for every protected outcome, not only the blended score. When objectives conflict persistently, the right response may be product redesign or less personalization rather than another optimizer.

Facebook once did that arithmetic in public. In the fourth quarter of 2017 it changed video recommendations to favour videos that encourage meaningful social interactions — a non-engagement objective. Then it priced the change, on the earnings call of 31 January 2018. Mark Zuckerberg put it plainly: “We estimate these updates decreased time spent on Facebook by roughly 5% in the fourth quarter.” He gave the same quantity again in the unit a user would recognise, “an estimated 50 million hours every day”.

He also said he expected the News Feed change announced that January to reduce time spent and some measures of engagement further. The company then filed that expectation with the SEC as a risk factor, in its Form 10-K of 1 February 2018: “we recently announced changes to our News Feed ranking algorithm to help our users have more meaningful social interactions, and we expect that these changes will have the effect of reducing time spent and some measures of user engagement with Facebook, which could adversely affect our financial results”. Announcing the change, Adam Mosseri had already named the mechanism: “Because space in News Feed is limited, showing more posts from friends and family and updates that spark conversation means we'll show less public content.”

Whatever one thinks of the change itself, that is the shape of the evidence a multi-objective ranker owes its reviewers. The objective that gained is named. The objective that paid is named. The payment is a number. And the number survives contact with a securities filing, where being wrong has consequences. A launch review that produces only a blended win has produced none of that.