Recommender systems
Baselines, Heuristics, and Popularity Recommenders
Build strong non-personalized, segment, recency, and rule-based baselines before attributing value to complex recommendation models.
By the end you can
- Construct a ladder of meaningful recommendation baselines
- Distinguish experimental comparators, production fallbacks, and editorial policies
- Use baseline outcomes to diagnose where recommendation value originates
- Design fair-budget comparisons and rollback-ready fallback policies
Example
The neural model lost to “trending this week”
A news article has about two days to live. Microsoft measured it. MIND, the news dataset it released in 2020, carries 1,000,000 users and 161,013 news articles in 20 categories. Behind those articles sit 15,777,377 impressions and 24,155,470 click behaviours from Microsoft News logs. Survival time is the gap between an article's first appearance in those logs and its last. “We find that the survival time of more than 84.5% news articles is less than two days.” Cold start is not an edge case in that inventory. The paper calls it “a common phenomenon in news recommendation”, and judges traditional ID-based recommenders unsuitable for the task.
That is the domain a news app is buying personalization for. One such app invested in a large personalized model. On new sessions a recency-weighted popularity list produced higher satisfaction and lower latency. The model was leaning on sparse profile history, against an item pool that mostly did not exist two days ago.
- Strong context signal: With more than 84.5% of MIND's articles surviving less than two days, current events dominate durable taste for many visits.
- Sparse identity: A large share of traffic was anonymous or newly registered — the condition the MIND authors call “a common phenomenon in news recommendation” when they judge ID-based recommenders unsuitable here.
- Serving cost: The complex model increased latency and reduced page completion.
- Evaluation flaw: The team compared against raw all-time popularity rather than a tuned baseline. That is one of the two ways a table gets decided before the models compete. The other is scoring against a sample of the catalogue instead of all of it, which Krichene and Rendle took apart in 2020.
- Diagnostic value: The failure showed that freshness and surface intent mattered more than long-term personalization.
Baselines expose whether personalization earns its complexity
Any new recommender has to beat the plausible alternatives under the same eligibility, latency, and evaluation protocol. Useful baselines are unglamorous ones: recent popularity, segment popularity, co-occurrence, editorial rules, business rules, random eligible items, and whatever policy is running in production today. A baseline is not merely a low bar. It reveals whether gains come from personalization, freshness, inventory, candidate recall, or extra compute.
The rest of this lesson is a series of cases in which someone measured the cheap alternative properly and the reported gain moved. A counting window worth 70% to 83%. A retuned matrix factorization worth five years of benchmark progress. A co-occurrence rule that has been the production policy since 1998.
Without the cheap alternatives measured under the same protocol, nobody can say which part of the system produced the gain.
Steps
A baseline suite, assembled before model selection
Random eligible items are the floor. The suite climbs from there through popularity windows, relational rules, and matched budgets. The last rung promotes one of these policies to a fallback the system can actually fall back to.
Two rungs have measured content behind them rather than taste. Step 2's four counting windows — all-time, recent, decayed, segment — are the ones separated on MovieLens 20M, where the gap between the first two ran from 70% to 83%. Step 4's matched budgets are what a 2021 comparison of twelve session-based techniques across eight datasets meant by identical conditions: same data, same tuning effort, same measured running time on both sides. The running-time column decided its verdict.
1. Implement random eligible
Use it to detect leakage and broken metrics.
2. Tune popularity windows
Compare all-time, recent, decayed, and segment counts.
3. Add relational rules
Test co-view, co-buy, complement, or editorial candidates.
4. Match budgets
Equalize eligibility, freshness, latency, and evaluation splits.
5. Promote a fallback
Operationalize the simplest policy that meets minimum quality and safety.
Visual
A practical baseline ladder
Climb the ladder in order. Random eligible items test whether the pipeline works at all. The current production policy is the number a new model actually has to beat. The rungs in between tell you which part of the gap is popularity and which part is personalization.
The step from global to recent popularity has been measured. Almost everyone defines the standard MostPop baseline the same way: raw interaction counts over the whole training set. Four researchers went looking and found that same answer in 12 papers from KDD, WWW, SIGIR and RecSys, and in 6 open-source toolkits. Then they re-ran it. The test was MovieLens 20M, leave-one-out over 29,431 test instances from 31 Mar 2010 to 31 Mar 2015. MostPop scored HR@10 0.0462 and NDCG@10 0.0248. RecentPop counts the same interactions over a one-month window ending at the user's own interaction time; it scored HR@10 0.0845 and NDCG@10 0.0440. DecayPop, exponentially decayed counts over six months, scored HR@10 0.0843 and NDCG@10 0.0441. Their own summary, from 2020: “First, RecentPop achieves significant improvement over MostPop on all evaluation metrics. The improvement is in the range of 70% to 83%.”
No model was added. Nothing was learned, personalized or embedded. The counting window moved, and the bar a new recommender has to clear rose by most of its own height.
- 01
Random eligible
Checks pipeline integrity and metric behavior.
- 02
Global popularity
Measures the strength of head-item demand.
- 03
Recent popularity
Adds time sensitivity and trend response.
- 04
Segment popularity
Uses coarse context such as market, device, or lifecycle stage.
- 05
Co-occurrence or rules
Adds item relations, editorial knowledge, or operational constraints.
- 06
Current production policy
Establishes the real replacement threshold.
Analogy
A thermometer before a weather model
Tested against nothing, a complex weather forecast always looks impressive. Tested against yesterday's temperature and the seasonal average, it either adds information or it does not. Recommendation baselines play that same diagnostic role. A personalized ranker that cannot beat trending this week has said something precise about where its gains were supposed to come from.
The analogy has a sharp edge. Yesterday's temperature is a fixed, agreed quantity. A recommendation baseline is something the experimenter builds, and can build badly. MostPop over the whole training set and RecentPop over a one-month window are both honestly called popularity, and they are 70% to 83% apart.
Complexity earns its place by beating simple, well-specified alternatives.
Example
Weak-baseline practices
An untuned straw man inflates the gain a new model appears to deliver. No fallback rehearsal means the simple policy that was supposed to catch an outage has never been run under one. Both are failures of the baseline, at opposite ends of its life.
- Untuned straw man: A deliberately poor comparator exaggerates model gains — the same Bayesian matrix factorization stands at RMSE 0.8197 in the published record, and at 0.7633 once Rendle and colleagues tuned it themselves.
- Unequal candidate sets: The complex model receives richer inventory or fresher data.
- Unequal latency budget: The baseline is forced through slower infrastructure or stale caches, and neither side's running time is reported.
- No segment analysis: Average gains hide losses for new users or sparse markets.
- No fallback rehearsal: The simple policy exists in code but fails during a real outage.
Key idea
No win over a tuned baseline, no incremental value
A sophisticated model that cannot beat a tuned, fair-budget baseline has not established incremental value. Tuned is the load-bearing word. On MovieLens 10M the distance between an untuned and a tuned Bayesian matrix factorization is RMSE 0.8197 against 0.7633. The distance between that tuned baseline and MRMA, the best method ever reported on the benchmark, is 0.7633 against 0.7634. The tuning gap is the larger of the two.
A model that only matches the tuned baseline has bought nothing but its own serving cost.
Baseline results guide architecture
If recent popularity dominates, invest in freshness and trend features. If item-item co-occurrence is strong, retrieval quality may matter more than a complex ranker. If personalization helps only after ten interactions, design onboarding and fallback states explicitly. Keep baseline dashboards after launch. When a complex model regresses below them, rollback becomes an evidence-based action rather than a political debate.
A winning simple policy is also a candidate for the production policy, not only a diagnostic. Amazon launched item-to-item collaborative filtering in 1998 — a co-occurrence rule reduced at serving time to a series of lookups. It scales to hundreds of millions of users and tens of millions of items without sampling, and updates immediately on new information about a customer. Smith and Linden of Amazon reported that in 2017. By then it had spread to YouTube, Netflix and much of the web, still competitive in perceived quality against the newer algorithms of the intervening two decades. “Because of its simplicity, scalability, explainability, adaptability, and relatively high-quality recommendations, item-based collaborative filtering remains one of the most popular recommendation algorithms today.” The oldest policy in the ladder is also the one still serving.
The baseline that wins points at where the next engineering effort belongs: freshness, retrieval, or onboarding.
Case
Eighteen neural methods, seven reproduced, six beaten by neighbours
The strongest evidence for baselines is a reproduction study. Eighteen neural top-n methods from recent conferences went into one, run by Ferrari Dacrema and colleagues, and seven came out reproducible. Six of those seven were often beaten by nearest-neighbour or graph heuristics. Google's own paper reached a comparable verdict from the other direction: with proper hyperparameter selection, a plain dot product substantially outperformed the learned similarity that neural collaborative filtering had popularised.
That first count did not stay a single result. The same group widened the sample to 26 papers published between 2015 and 2018 at KDD, IJCAI, WWW, SIGIR, RecSys and WSDM. “We scanned the proceedings of several conference series and identified 26 relevant papers. We could reproduce 12 (46%) of them with reasonable effort.” Of those 12, 11 were outperformed by conceptually simple methods such as nearest-neighbour heuristics, and 8 by nearest neighbours alone. One was clearly better than the non-neural baselines, and that on only one of the datasets considered. Reproducibility by venue ran from 3 of 4 at KDD to 1 of 7 at RecSys and 0 of 1 at WSDM.
Tuning the baseline is part of the experiment.
Figure
Position
The comparison decides more of the result than the architecture
Eighteen neural top-n methods went into that first review and seven came out reproducible with reasonable effort. The larger follow-up scanned 26 and reproduced 12 (46%). Six of the original seven could often be outperformed by nearest-neighbour or graph-based heuristics; 11 of the later 12 were outperformed by conceptually simple methods. The qualifiers are the finding, not the counts. Often means these methods sometimes win. Reproducible with reasonable effort is a statement about what the papers made available, not a verdict on the ideas inside them.
The comparator has a price, and someone has put a number on it. Five years of published results on the MovieLens 10M rating-prediction benchmark rested on untuned baselines. Rendle and two colleagues at Google showed that in 2019. A Bayesian matrix factorization they tuned themselves reached RMSE 0.7633. The same model, BPMF, stands at 0.8197 in the literature. MRMA, the best method ever reported on the benchmark, stands at 0.7634. Adding decade-old SVD++/timeSVD++ techniques took the tuned baseline to 0.7485, beating the previous best published result by 0.0144. The entire 2013-2017 progress on that dataset, from LLORMA at 0.7815 to MRMA at 0.7634, was 0.0186. “Instead of improving over the baselines by a large margin, all recently proposed methods underperform well-known baselines substantially.” Careful tuning of an old model outran five years of new ones. Rendle and colleagues found the same shape from the other direction when they revisited neural collaborative filtering: with proper hyperparameter selection, a simple dot product substantially outperformed the learned similarities that architecture had popularised.
Tuning is not the only lever that settles a comparison before any architecture competes. Scoring a recommender against a small random sample of items rather than the full catalogue is not merely noisier. It can reverse the answer. Krichene and Rendle showed that in 2020. “This paper investigates sampled metrics in more detail and shows that they are inconsistent with their exact version, in the sense that they do not persist relative statements, e.g., recommender A is better than B, not even in expectation.” As the sample shrinks, all metrics collapse toward AUC. Their conclusion is that sampling should be avoided for metric calculation. For studies that must sample anyway, they derive bias- and MSE-minimising corrections.
None of this says simple models are better in general. It says a published comparison can be settled by how much care each side was given and by how the metric was computed, and a headline number carries neither. So the question to put to any recommender claim is not which architecture won. It is what the losing side was allowed to do, and over how much of the catalogue it was scored.
The strength of the comparator is part of the result, and a headline number does not carry it.
Key idea
The complexity gate
Add personalization only after the system beats strong simple policies on value, guardrails, and operational cost. That gate has been run in public. Twelve session-based recommendation techniques were compared under identical conditions across eight datasets. Six of them were recent neural methods, published at outlets including ICLR, CIKM, KDD, WSDM, AAAI and SIGIR. In most experiment configurations one of the simple techniques outperformed all the neural approaches. Across the four e-commerce datasets, only on RSC15 did a neural method, narm, slightly outperform the best simple baseline, vstan — and only on MAP, Precision and Recall. On the music dataset 8TRACKS narm also took the best Precision and MRR. Differences between the best neural and non-neural methods were “in most cases around or less than 1 %”.
Then the cost of that sub-1% win. On ZALANDO the best neural method, sr-gnn, needed 1,527.17 minutes per split to train. vstan needed 0.13. vstan also recommended in half the time, 55.58 ms against 120.15 ms. The authors leave the question where it belongs: “In general, even in case where a complex neural method would slightly outperform one of the more simple ones in an offline evaluation, it remains open if it is worth the effort to put such complex methods into production.”
Personalization that clears only the value bar and not the guardrail or cost bar has failed the gate, not passed two-thirds of it.
Key takeaways
- Simple recommenders are not embarrassing; they are the controls that make complex claims credible — of the 12 papers the larger review could reproduce, 11 were outperformed by conceptually simple methods and 8 by nearest neighbours alone.
- Any new recommender has to beat the plausible alternatives under the same eligibility, latency, and evaluation protocol. That is what running twelve session-based techniques under identical conditions across eight datasets was for.
- A sophisticated model that cannot beat a tuned, fair-budget baseline has not established incremental value. On MovieLens 10M the same Bayesian matrix factorization moves from RMSE 0.8197 in the literature to 0.7633 once it is tuned properly — and MRMA, the best method ever reported there, sits at 0.7634.
- Random eligible items and metric hygiene come before every other rung, because a sampled metric is inconsistent with its exact version and does not preserve “recommender A is better than B”, not even in expectation.
- The untuned straw man remains a practical risk. With decade-old SVD++/timeSVD++ techniques added, that tuned baseline reached 0.7485 and beat the previous best published result by 0.0144, against 0.0186 for the whole 2013-2017 progress on the benchmark.
- The simplest policy that clears minimum quality and safety deserves to run in production as a fallback, not merely to sit in an evaluation table — Amazon's item-to-item collaborative filtering launched in 1998 and was still, in 2017, one of the most popular recommendation algorithms in use.