Ranking, Scoring, and Structured Prediction

A search system must decide which candidates appear and where they appear. In the European Union those decisions are a legal object as well as an engineering one. The Platform-to-Business Regulation, Regulation (EU) 2019/1150, has applied since 12 July 2020. It defines 'ranking' as the relative prominence given to goods or services, and Article 5(1) says what a platform owes the businesses it ranks: “Providers of online intermediation services shall set out in their terms and conditions the main parameters determining ranking and the reasons for the relative importance of those main parameters as opposed to other parameters.” Article 5(3) adds disclosure wherever remuneration can influence ranking. The Commission issued ranking transparency guidelines on 7 December 2020.

The disclosures exist. Booking.com's own published description of how it works names click-through rate, gross bookings and net bookings as its ranking areas. It also states that commission paid, payment speed and programme membership can influence an accommodation's ranking. The rules layer around a ranking model is not a rumour about search systems. Here it is written down twice: once in a statute, once in a platform's own terms.

Not every target behaves alike. Two of them can both be called “prediction” and still need very different learning and evaluation logic.

Scoring assigns a numeric value to each item independently, then a threshold converts it to action — a fraud risk score is the familiar case. It is useful for prioritization and blind to competition among items. Ranking produces an ordered list for a context: relative order is central, top positions carry more value, and exposure creates biased feedback. Sequence labeling predicts linked labels along an ordered input, so neighbouring outputs affect coherence, errors propagate, and evaluation can be token- or span-level. A set or assignment output chooses a collection under global constraints, where capacity and exclusivity matter, independent scores can conflict, and optimization follows prediction.

Scoring looks like the mildest of the four. It carries the largest documented failure in this lesson. A commercial risk-scoring tool is applied to roughly 200 million people in the United States each year. It auto-enrolled patients above the 97th percentile of its score and referred those above the 55th percentile to a physician. It had been trained to predict health-care cost rather than illness. At the 97th-percentile threshold, Black patients carried 26.3% more chronic conditions than White patients — 4.8 against 3.8. Obermeyer and three colleagues put the size of the correction in one line of their abstract, in Science on 25 October 2019: “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” On the day the paper appeared, New York's Department of Financial Services and Department of Health wrote to UnitedHealth Group about the Optum 'Impact Pro' algorithm.

No step of that pipeline was exotic. A score was produced, two cut-offs were applied, and the choice of what the score predicted decided who got help.

In ranking, the question is usually whether a useful item appears above a less useful one. A pointwise classifier can help without settling that. The final metric may emphasize order near the top.

That is measurable on a public benchmark. The TREC 2019 Deep Learning Track, run by NIST with Microsoft and UCL, evaluated ranking on the MS MARCO collections. The document task held 3.2 million documents with 367 thousand training queries. The track's overview paper describes the other one: “The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test set of 43 queries.” 15 groups submitted 75 runs, judged blind against reusable test sets of 43 queries each. The MS MARCO dataset underneath, published by Microsoft in 2016, holds 1,010,916 anonymized questions sampled from Bing query logs and 8,841,823 passages extracted from 3,563,535 web documents.

Hold those two numbers together: 8.8 million passages to learn from, 43 queries to be judged on. The judgments are expensive because a human has to read the top of each returned list. The top of the list is where the metric lives. Training scale is not evaluation scale, and the thing being scored is an order, not a pile of independent decisions.

Structured tasks add dependencies or constraints among outputs. Predict each component separately and you can produce impossible sequences, duplicate assignments, or combinations that violate policy.

A ranking system influences which outcomes it later observes. Policy and data form a closed loop. A first-stage retriever limits what can be ranked at all. The model then scores and orders those candidates according to learned and rule-based signals. The rendered positions allocate exposure, and higher positions receive more attention regardless of intrinsic relevance. Clicks, dwell time, purchases and skips are recorded selectively, only for what was shown. The next model trains on that record. It may amplify the patterns the previous ranking created.

Write the output contract first. A precise one prevents a ranking or structured task from collapsing into vague scoring. Name the unit — the query, sequence, session, batch, or assignment context. State the constraints: exclusivity, capacity, order, coverage, safety. Trace the feedback, identifying which outputs receive labels or interaction evidence and why. Choose the objective, deciding whether pointwise accuracy, pairwise order, or whole-list quality matters. Then evaluate the structure: top positions, complete spans, feasible assignments, important slices.

What a loose contract costs has been measured on a benchmark in use since 2003. In 2019 six authors re-checked the CoNLL-2003 English named entity recognition test set and reported the damage: “In the end, we have corrected label mistakes in 186 sentences, which is about 5.38% of the test set.” State-of-the-art F1 at the time was around 93%. A second team re-annotated the same test set independently in 2024, correcting 457 token labels plus 63 sentence-boundary repairs. Models scored far higher on the cleaned set: XLM-R FLERT rose from 93.64 to 95.98 F1, and LUKE from 94.44 to 97.10.

Part of that reported distance was disagreement about where a span begins and what it is — the output contract — rather than model quality. A benchmark that has been scored for two decades can still be arguing with itself about its own labels.

A learned model may estimate relevance, duration, risk, or compatibility, while a downstream optimizer assembles the final list or assignment. The separation can make constraints explicit and simplify auditing.

The US residency match runs that separation at national scale. Applicants and programs submit rank order lists — the preference stage. Then one algorithm produces a stable assignment subject to constraints such as couples matching to paired positions. In the 2025 Main Residency Match, 52,498 applicants registered against 43,237 certified positions. The National Resident Matching Program published the outcome on 21 March 2025: “Of the 43,237 training positions offered, 94.3 percent (40,764) filled when the matching algorithm was processed.” Roth and Peranson redesigned the algorithm in use in 1999, precisely because the earlier one could be manipulated and could not handle complementarities such as couples. The couples constraint is stated in the algorithm, not learned from data. That is why it can be examined at all.

Do not credit the model alone for system behavior. Candidate generation, business rules, optimization, and interface design can dominate the observed result.