Capstone: Design and Defend a Recommendation System

Five rows, and a defensible answer fills every one. Candidates want to start at the recommendation architecture row. The row above it decides whether that architecture was the right thing to build at all.

Two of the five rows now answer to someone outside the team. Problem and policy answers to the EU's Digital Services Act, which has applied since 17 February 2024. The Act fixes what a recommender must disclose and what option it must offer. The operating model answers to MANAGE 2.4 of the NIST AI Risk Management Framework, published in 2023. That is where the stop-and-retire plan gets graded against an external standard instead of against the team's taste. The three rows between them — evidence, architecture, evaluation — are graded against published results. The sections below name which ones.

A regional marketplace with 800,000 active listings wants a new home feed. Users include anonymous visitors, returning buyers, professional purchasers, and households. Sellers range from individuals to large retailers, inventory changes rapidly, and some categories carry safety or age restrictions.

The hard part of this brief is not the feed. It is the log the feed will be built from. That difficulty is not particular to this marketplace. Schnabel and Joachims gave the general statement of it in 2016: “Most data for evaluating and training recommender systems is subject to selection biases, either through self-selection by the users or through the actions of the recommendation system itself.” Their answer was to treat a recommendation as a treatment. Propensity-based estimators, adapted from causal inference, then give unbiased performance estimates from data collected under a previous recommendation policy. That is what a capstone team cites when it says what it can and cannot recover from this marketplace's history. A team that cannot name a method for it has not designed an evidence system at all.

Students must translate the product request into a surface contract, entity schema, exposure log, label and outcome design, candidate portfolio, ranking policy, evaluation plan, experiment, monitoring system, and lifecycle controls.

Two lines of that contract are no longer the team's to choose. The Digital Services Act was adopted on 19 October 2022 and has applied since 17 February 2024. Article 27(1) requires online platforms to set out the main parameters of their recommender systems in plain and intelligible language. Article 38, headed “Recommender systems”, goes further for the largest services: “In addition to the requirements set out in Article 27, providers of very large online platforms and of very large online search engines that use recommender systems shall provide at least one option for each of their recommender systems which is not based on profiling as defined in Article 4, point (4), of Regulation (EU) 2016/679.” A non-profiling option and published ranking parameters have article numbers and a date. They are not design preferences a reviewer can be talked out of.

What a student finally recommends may be a sophisticated multi-stage system, a limited hybrid, a non-personalized baseline for some states, or a decision not to deploy at all when the evidence and the controls are inadequate. Note what has happened to the third of those. For a service of that size, the non-personalized baseline is no longer only a fallback the team chose. It is a thing the law already obliges the surface to carry.

The register runs from objective risk to operational risk, and a capstone defense is judged on both ends. Clicks and purchases may not stand for satisfaction, trust or marketplace health. Stale indexes, inventory, experiments and fallbacks can invalidate a slate that was correct when it was computed.

One entry in this register is not a worry but a proved result. Score a recommender against the relevant items plus a small random sample of the others — the standard shortcut — and the score can lie about which system is better. Krichene and Rendle showed it at Google in 2020: “This paper investigates sampled metrics in more detail and shows that they are inconsistent with their exact version, in the sense that they do not persist relative statements, e.g., recommender A is better than B, not even in expectation. Moreover, the smaller the sampling size, the less difference there is between metrics, and for very small sampling size, all metrics collapse to the AUC metric.” The failure is not that a sampled metric is noisy. It is that the metric can hand back the wrong ordering of two systems. And a small enough sample flattens every metric the team thought it was choosing between into one.

The product contract is step one: surface, choice unit, outcomes, non-goals and stop rules. The decision is step six — progressive deployment, limited scope, more research, or rejection. The four steps between them are evidence, architecture, evaluation, and operations. A capstone that opens on architecture has skipped the part that makes everything after it defensible. Step two specifies exposure-aware logs, time, identity, labels, privacy and data gates. Step three compares baselines, hybrid sources, ranking, re-ranking and fallbacks.

Step four builds the evaluation ladder. Its last rung — long-term follow-up, after the experiment has already read out — is the rung teams cut. Google built experiment methodology for measuring long-term user learning rather than short-term treatment effects. It quantified ads blindness and sightedness on Google search, and in 2015 it reported what it did with the result: “We describe two major applications of our findings: a conceptual change to our search ads auction that further increased the importance of ads quality, and a 50% reduction of the ad load on Google's mobile search interface.” A short-horizon metric and a long-horizon one disagreed by enough that the company cut its own mobile ad load in half. That is the size of the gap a capstone leaves unmeasured when the evaluation program ends at the experiment.

Step five defines operations and governance: SLOs, monitoring, user and provider controls, incident ownership, and retirement. It has a numbered external referent. Subcategory MANAGE 2.4 of the NIST AI Risk Management Framework, published in 2023, reads: “Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use.” Two nouns in that sentence are what capstone submissions usually miss. Mechanisms that are applied, not merely available. Responsibilities assigned to a person, not to the team in general.

The submission should include a system diagram, surface contract, entity and exposure schema, baseline table, candidate-source analysis, ranking objective, evaluation protocol, experiment design, fairness and privacy review, monitoring dashboard, incident scenario, and retirement checklist.

Reviewers should challenge the weakest link rather than admire the most complex model. There is a measurement of what happens when a field does the opposite. A 2019 replication re-ran the recent neural top-n recommendation literature: “Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques.” The seventh did not consistently beat a well-tuned non-neural linear ranking method. Eighteen published systems. Seven reproducible. Six of those seven beaten by nearest-neighbour or graph-based heuristics. That is the arithmetic the baseline table exists to catch before a capstone repeats it. A well-tuned simple baseline is a rival, not a formality.

The capstone passes when another team can reproduce the reasoning, identify uncertainties, and stop the system when assumptions fail.