Calibration and Preference Consistency

A user has watched 70 romance movies and 30 action movies. The list that user should get back is about 70% romance and 30% action. The list an accuracy-optimised ranker returns instead is romance in every slot. Each romance candidate outscores each action candidate, and nothing in a score-ordered top-N ever inspects the proportions of what it has assembled. Steck set the problem out in 2018 with exactly this user. His warning is precise: a system tuned for accuracy alone lets a user's lesser interests get crowded out. The thirty action films are not outvoted. They are never counted.

The two calibrations share a word and almost nothing else. Profile calibration is a property of the list: does its composition resemble the user's interest mix? Probability calibration is a property of the score: does a predicted probability match how often the outcome actually happens? The second is measured as expected calibration error, a score-level statistic that depends on population and time. It says nothing whatever about what a list contains.

Intent-aware diversification is the third option, and it does not have to be argued about. Somebody put it on a bench beside calibrated re-ranking. Kaya and Bridge ran that comparison in 2019, against the intent-aware methods xQuAD and SPAD, on two datasets. MovieLens 20M after preprocessing: 137,765 users, 20,631 items, 9,990,460 interactions. The Taste Profile subset: 375,749 users, 190,629 items, 19,328,665 interactions. Their section on the distinction opens on the distinction itself: “Intent-aware recommendation may result in calibrated recommendations, but it does not directly aim to cover the user’s interests in the same proportion as they occur in the user’s profile.” That is the whole difference in one sentence. Covering plausible sub-intents is not the same objective as reproducing proportions. A method can deliver the first and land anywhere on the second. What the comparison reported: defining a user's interests as subprofiles rather than as item features gave the highest precision and the best relevance/diversity trade-off.

Calibration compares two distributions, and both halves of the comparison are built rather than found. Profile construction estimates preference proportions from the user's history and explicit evidence. Attribute representation chooses the dimensions the proportions are taken over — genres, topics, price bands, creators. Then the composition of the candidate or final slate is measured on those same dimensions.

The fourth step has a standard form. The degree of calibration C_KL(p,q) is the Kullback-Leibler divergence between p(f|u), the distribution of a feature over the items the user has liked, and q(f|u), the distribution of that feature over the recommendation list. KL has a sharp edge. It diverges whenever q(f|u) is 0 while p(f|u) is greater than 0. That is a genre the user actually watches, missing from the list entirely — the exact case the metric exists to punish. Rather than let the measure blow up, q is replaced by a smoothed q̃ = (1-α)q(f|u) + αp(f|u). The constant is not derived. It is inherited, and Kaya and Bridge say so outright: “Steck uses α = 0.01, and we do the same.” One hundredth of the user's own profile is poured into the list's distribution before the divergence is taken. So a list that contains none of an interest is never scored as containing none of it. Steck also proves that the combined relevance-plus-calibration objective satisfies the conditions under which greedy re-ranking attains a 1 - 1/e approximation to the optimum. A real guarantee, and a bounded one.

The last step is the one teams skip: asking the user and the context whether that target mix is wanted right now. No divergence, however carefully smoothed, contains that question.

In recommender systems, calibration usually means matching the list to the profile. The distribution of properties in the recommended list should resemble the distribution in the user's history. In practice that is a Kullback-Leibler divergence between a feature distribution over the items a user liked and the same feature distribution over the list, smoothed by α = 0.01 so it stays finite. A calibrated list can preserve multiple interests and reduce systematic overrepresentation. Probability calibration is a different thing entirely. There a predicted probability should match the frequency actually observed, and the measurement is an expected calibration error computed over M = 15 bins. Both can hold at once. Each is measured its own way, on its own object: one on a list of items, one on a number attached to a single prediction.

Show tradeoff curves between relevance and composition mismatch. Evaluate users with narrow, broad, changing and sparse profiles separately, because the trade does not land the same way on all of them. That is a measured claim, not a caution.

Spotify measured it in 2020. The data covered over 100 million distinct users who streamed around 70 billion times in the first 28 days of July 2019, and the analysis itself was restricted to premium users. Each user's diversity was scored with a generalist-specialist score, the GS-score. Algorithm-driven listening came out less diverse than user-driven listening. Profile shape then tracked the outcomes a platform actually watches: “Controlling for activity (the number of songs played), generalist users are up to 25 percentage points less likely to leave the platform and up to 35 percentage points more likely to become a paid subscriber.” A randomized experiment in the same work found that relevance mattered more to specialists than to generalists. The same relevance-for-composition trade, priced differently for two kinds of user inside one product.

So offer modes or controls where the product supports them. “More like this” can intentionally narrow the distribution; a home feed may preserve a wider preference mix.

Attribute choice bias happens before anything is calibrated. A taxonomy nobody chose for this purpose decides which preferences are allowed to count. The attribute space of the published calibration literature is largely inherited: “In the original ML20M dataset, all users have at least 20 movies in their profile and movies have one or more of 19 genres.” Every proportion, every divergence and every claim of a well-calibrated list is a statement about those labels and nothing else.

Miscalibration is also not spread evenly across users. A 2020 study took miscalibration as the measure of how far a recommender's output sits from a user's true preference distribution. It reported from two real-world datasets: “Moreover, we show that the more a group is affected by the algorithmic popularity bias, the more their recommendations are miscalibrated.” The users furthest from popular taste are the ones served worst. An average divergence hides them.

False comfort happens after all of it, when a calibrated list gets described as fair, satisfying or diverse without separate evidence for any of the three.

A calibrated policy starts from the profile window, keeping durable, recent and explicit interests apart. The published formulations skip that step: both the item weights and the rank weights are set to 1. Second, choose attributes that are meaningful, governable and sufficiently complete — knowing that in most reported results the attribute set is 18 named genres plus “(no genres listed)”. Third, select the mismatch measure. That is the Kullback-Leibler divergence between p(f|u) and q(f|u), with the smoothing constant stated out loud rather than inherited. α = 0.01 is a choice about how much of the profile is lent to the list before the comparison is made.

The fourth step, re-ranking within value limits, is usually treated as one obvious move. It is a solver choice with published consequences. Greedy re-ranking comes with the 1 - 1/e approximation guarantee. A 2021 paper spelled out what that leaves on the table: “Greedy heuristic approaches have been proposed to calibrate recommendations, and although they provide great improvements, they can result in inefficient solutions in that a better one can be missed because of the myopic nature of these algorithms.” Its constrained-optimization model, combining accuracy and calibration, outperformed the state-of-the-art heuristics in most cases — on both the accuracy of the recommendations and the level of calibration achieved.

The fifth step ends at user outcomes rather than at distribution distance. Satisfaction, discovery, control and segment effects are the test that matters. The Spotify work measured churn and conversion across generalists and specialists, not the distance between two histograms.