Reranking and Context Assembly

The first retriever is usually optimized for speed and candidate recall. Its top results can contain duplicates, weak paraphrases, stale versions, and passages that mention the query without answering it.

That this output is an intermediate artifact rather than an answer is a benchmark result, not a design opinion. BEIR tested 10 retrieval systems across 18 publicly available datasets in 2021, with no in-domain training. One sentence in its abstract carries the finding: “Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs.”

Read the two halves of that sentence together. A lexical baseline is hard to beat out of domain. The systems that do beat it on average are the ones that add a second stage on top of the candidate set. That stage is bought with compute, per query, at serving time.

Reranking and assembly decide what the generator will actually see. The stage can rescue a noisy candidate set. It can also throw away the only decisive passage.

The candidate budget and the latency target determine which methods are practical. Four families are in common use, and three of them have a published number attached.

Heuristic reranking uses metadata, authority, freshness, source diversity, and explicit rules. It is fast, explainable, and the right place to encode business policy. It has no semantic interaction with the query, so its usual role is to shrink or pre-filter the candidate set before an expensive model runs.

Cross-encoder reranking jointly encodes query and passage, so every query token can attend to every passage token. Nogueira and Cho measured what that buys in 2019 by running a BERT cross-encoder over query–passage pairs: “Our system is the state of the art on the TREC-CAR dataset and the top entry in the leaderboard of the MS MARCO passage retrieval task, outperforming the previous state of the art by 27% (relative) in MRR@10.” That 27% relative gain in MRR@10 is a reordering gain. The reranker fetched nothing. It only changed which of the already-retrieved passages came first. The cost is that the model runs once per query–passage pair, so latency grows with the candidate count, and it needs labels shaped like the task.

Late-interaction or multi-vector scoring keeps finer-grained document representations, so most of the encoding can be done offline. ColBERT introduced it in 2020, and its MS MARCO re-ranking table gives the cost curve directly: MRR@10 34.9 at 61 ms per query, against BERT-base's 34.7 at 10,700 ms. Khattab and Zaharia state the trade in their abstract: “Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query.” Equal effectiveness, two orders of magnitude of latency removed. It is paid for in a larger index and more serving complexity.

LLM-assisted selection asks a generative model to judge, rank, or extract evidence from candidates. It works: Sun and colleagues showed at EMNLP 2023 that “properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods” on passage relevance ranking for popular IR benchmarks. The flexibility is real and so is the exposure. Candidate text is untrusted input even when it is only being scored. OWASP ranks prompt injection as LLM01 in its Top 10 for LLM Applications 2025 and defines the indirect form precisely: “Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files. The content may have in the external content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways.” NIST gave the same class a place in the federal adversarial machine learning taxonomy in March 2025. A reranker that reads passages needs constrained inputs, narrow outputs, and policy checks outside the model.

The objective includes more than relevance score, and five constraints compete for the same window.

Evidence coverage comes first. The package must contain the passages needed to answer every material part of the request, not the passages that scored highest on the whole of it. Authority and freshness come next: controlling, current, permitted sources preferred over popular ones, which requires that the metadata survive chunking. Token efficiency sets the ceiling — enough surrounding context for each passage to be interpretable, without spending the window on one verbose document. Generator usability is the presentation layer. Order and label the evidence so that roles, dates, and provenance stay legible to both the model and the evaluator.

Diversity and contradiction is the layer with the oldest named mechanism, and naming it matters, because it is usually implemented without being examined. Maximal Marginal Relevance comes from Carbonell and Goldstein, in a two-page poster from 1998. Its criterion: “The Maximal Marginal Relevance (MMR) criterion strives to reduce redundancy while maintaining query relevance in re-ranking retrieved documents and in selecting appropriate passages for text summarization.”

Two things follow for assembly. The criterion trades query relevance against novelty, so it is a knob with a direction, not a neutral cleanup pass. And its clearest measured advantage was in building non-redundant multi-document summaries — a task where dropping a near-duplicate is always correct. Evidence assembly is not that task. A later correction can be textually near-identical to the claim it overturns. That is exactly the shape MMR is built to suppress.

The most instructive assembly failure has been measured. Cuconasu and colleagues varied passage relevance, position and count in The Power of Noise, published in 2024, and their result inverts the intuitive assembly rule: “One counter-intuitive finding of this work is that the retriever's highest-scoring documents that are not directly relevant to the query (e.g., do not contain the answer) negatively impact the effectiveness of the LLM. Even more surprising, we discovered that adding random documents in the prompt improves the LLM accuracy by up to 35%.”

The damaging passages are not the obviously irrelevant ones. They are the near misses that the retriever ranks at the top. So "take the top-k by score" is empirically the wrong assembly rule, and an accuracy improvement of up to 35% from random documents shows how far a ranking signal is from a selection criterion.

Use answer-bearing and contradiction-aware tests, in six steps.

First, freeze the candidate set, so that reranking is evaluated without upstream recall moving underneath it. Second, label passage utility, distinguishing decisive, supporting, redundant, contradictory, and irrelevant items. The Cuconasu result is only visible if "high-scoring but not answer-bearing" is a label you can count.

Third, compare rankers on gain, latency, calibration, and failure slices — and do it on a candidate set someone else can download. MS MARCO is the set on which the cross-encoder and late-interaction numbers above were measured. The 2018 version of its paper gives the scale: “In addition, the dataset contains 8,841,823 passages---extracted from 3,563,535 web documents retrieved by Bing---that provide the information necessary for curating the natural language answers.” That same version reports 1,010,916 anonymized questions sampled from Bing's search query logs, and the third of its three proposed tasks is explicitly to rank a set of retrieved passages given a question. Cite the version you used. The 2016 original announced the dataset, but it contains none of these numbers and describes no ranking task.

Fourth, test context budgets by varying chunk count, parent size, and per-source allocation. Fifth, sweep order, moving decisive and contradictory passages through the assembled context as Liu and colleagues did. Sixth, score generator use — claim support, omissions, abstention, and citation alignment — because a ranking metric measured at step three will not tell you whether the assembled package was usable.

A strong first-stage retriever maximizes the chance that useful evidence enters the candidate pool. Reranking and assembly decide what happens to it. BEIR located the best average zero-shot quality in exactly this stage, at high computational cost. Nogueira and Cho got 27% relative in MRR@10 out of reordering alone. ColBERT showed the same effectiveness could be had at 61 ms per query rather than 10,700 ms.

Then Cuconasu and colleagues showed that the ranking score which produces those gains is not, by itself, a selection rule. The retriever's own top non-answer-bearing passages hurt the generator, while random documents raised accuracy by up to 35%. A good score tells you a passage resembles the question. It does not tell you the passage answers it.

The next lesson focuses on grounding and citations. Once context is assembled, the generator must constrain claims to what the evidence actually supports.