Dense Retrieval and Neural Reranking

A dense retriever may understand that “appeal a delayed-flight reimbursement” relates to compensation policy. It can still miss the document “Form EU-261-B” when the query contains that exact identifier. A lexical index will find the code and miss the paraphrased need.

Hybrid systems exist because lexical identity and semantic resemblance are complementary signals. The design must preserve both rather than declare one universally superior.

This lesson makes that case from measurements rather than from taste. Margins of 29.4% and 37.4% NDCG@10 for neural rankers in the first blind TREC evaluation of them. MRR@10 34.9 at 61 ms against 34.7 at 10,700 ms for two architectures on the same candidate list. An index of 20 GB against 0.4 GB for the same million documents. A six-point MRR@10 collapse when hard negatives are mined without human review.

The query and document can be encoded separately. That is what enables large-scale indexing: the document side is computed once and reused across every query. It is also what makes it feasible to search all 8.8M passages of MS MARCO end to end, rather than reranking a candidate list someone else produced.

Two architectures reached the same ranking quality on MS MARCO. One answered in 61 ms and the other in 10,700 ms. That is the trade at issue here — expressiveness against the ability to precompute documents — and it has been measured on both axes.

The measurement is ColBERT, 2020. Re-ranking BM25's official top-1000, a BERT-base cross-encoder scored MRR@10 34.7 at 10,700 ms and 97 trillion FLOPs per query. Khattab and Zaharia's late-interaction model scored 34.9 at 61 ms and 7 billion FLOPs. The paper states the gap as “over 170x in latency and 13,900x in FLOPs”, and its abstract puts the claim plainly: “Results show that ColBERT's effectiveness is competitive with existing BERT-based models (and outperforms every non-BERT baseline), while executing two orders-of-magnitude faster and requiring four orders-of-magnitude fewer FLOPs per query.” Run end to end over the full 8.8M-passage collection instead of over a supplied list, ColBERT reached MRR@10 36.0 and Recall@1000 96.8, against Anserini BM25's 18.7 and 85.7.

The BEIR benchmark timed the same families independently in 2021, on 1M DBPedia documents: BM25 20 ms on CPU, dense bi-encoders 14-20 ms, ColBERT 350 ms, and BM25 with a cross-encoder reranker 450 ms on GPU and 6100 ms on CPU.

Storage is where the late-interaction row pays for its expressiveness. On those same million DBPedia documents the index was 0.4 GB for BM25, 3 GB for the 768-dimensional dense bi-encoders DPR, ANCE and TAS-B, 12 GB for SPARTA and 20 GB for ColBERT. Scale it up and the ratio holds: “ColBERT requires ~900GB to store the BioASQ (~15M documents) index, whereas BM25 only requires 18GB.”

ColBERT's own paper measures the knob that buys some of that back. Holding MS MARCO's 8.8M passages at 128 dimensions and 4 bytes per dimension takes 286 GiB. At 24 dimensions and 2-byte floats it takes 27 GiB, for a 1-point MRR@10 loss, 34.9 to 33.9.

A cost ordering is not the point. Each row of this comparison is a number somebody published, and the gaps are wide enough that the choice is rarely close once the corpus size and the latency budget are named.

Randomly unrelated documents teach an easy problem. They can also hide the confusions the product actually has.

Hybrid retrieval should expose what each channel contributed and where it failed. Step 3 has a default, and the default has a name and a published margin.

The name is Reciprocal Rank Fusion, and the method is one line: score a document as the sum over systems of 1/(k + rank). Cormack and colleagues fixed k = 60 in a pilot in 2009 and never retuned it. Their abstract: “Reciprocal Rank Fusion (RRF), a simple method for combining the document rankings from multiple IR systems, consistently yields better results than any individual system, and better results than the standard method Condorcet Fuse.” Across four TREC collections it beat Condorcet Fuse, CombMNZ and the best individual system by 4% to 5% MAP. On TREC Robust, RRF scored .3686 against .3586 for the best individual system, .3652 for Condorcet and .3575 for CombMNZ. Their k sweep is flat enough to explain why the constant survived: MAP .2072 at k=0, .2145 at k=60, .2098 at k=500.

That constant is what production systems ship today. Azure AI Search fuses BM25 and HNSW result sets this way, and Microsoft's documentation gives the formula: “The score is calculated as 1/(rank + k) ... Experiments show the algorithm performs best when you set k to a small value, such as 60”. Elasticsearch ships the same scoring loop with a default rank_constant of 60, links the 2009 paper, and states that “RRF requires no tuning, and the different relevance indicators do not have to be related to each other to achieve high-quality results”.

Convenience at step 3 is not permission to skip step 5. A fused list dissolves the evidence of which channel found each relevant item. Log it before you fuse.

Vector indexes can accelerate nearest-neighbor retrieval by searching a structured subset of the space. Recall then depends on index type, build settings, query settings, vector distribution, filtering, and hardware.

How wide is that spread in practice? One 2019 measurement comes from the paper introducing DiskANN: “On the billion point SIFT1B bigann dataset, DiskANN serves > 5000 queries a second with < 3ms mean latency and 95%+ 1-recall@1 on a 16 core machine, where state-of-the-art billion-point ANNS algorithms with similar memory footprint like FAISS and IVFOADC+G+P plateau at around 50% 1-recall@1.” The machine had 64 GB of RAM and an SSD. Microsoft Research repeats that configuration on its own publication page for the system.

Read the figure for exactly what it is: the authors' own 2019 evaluation of their own index, not an independently replicated result. Read its shape for what it means anyway. On one corpus, at a comparable memory budget, the choice of index structure moved 1-recall@1 from roughly half to over ninety-five per cent. Nothing about the encoder changed.

Measure index recall against exact search on a controlled sample. Include update, deletion, tenant filtering, memory, build time, and stale-vector behavior in the production evaluation.

Recall at candidate depth tells whether relevant evidence survives the first stage. Reciprocal rank, average precision, or discounted gain assess ordering under binary or graded relevance. Reranker metrics should be conditioned on its candidate set.

The first TREC-style blind evaluation of neural against traditional ranking shows why one number cannot serve both stages. NIST ran the TREC 2019 Deep Learning Track with Microsoft and UCL. Fifteen groups submitted 75 runs over a 3.2M-document and an 8.8M-passage corpus, with 43 judged test queries each. The organisers' overview opens on the headline result: “Deep learning runs significantly outperformed traditional IR runs.” The best pretrained-language-model run beat the best traditional run by 29.4% NDCG@10 on documents and 37.4% on passages. idst_bert_p1 scored 0.7645 where the best traditional passage run scored 0.5558, and it won on 36 of the 43 document queries.

Then the same overview separates the stages, and the margin changes size. Groups could either rerank the organisers' supplied candidates — top-100 for documents, top-1000 for passages — or retrieve end to end. Full retrieval beat pure reranking by 0.9% on documents and 3.6% on passages, even though the best passage run's NCG@1000 sat 20.7% above the provided candidate set. A 37.4% ranking gain and a 3.6% end-to-end gain are two true statements about one system, and they mean different things. The reranker was doing most of the visible work. The candidate stage it inherited was already good enough that replacing it bought little. Report only one of the two and the reader cannot tell which stage to invest in.

One of the sceptics recanted in print. Jimmy Lin wrote in SIGIR Forum in December 2019: “I now believe that pretrained transformer models are unequivocally more effective than those baselines, as well as pre-BERT neural models, in the low-resource regime”. He noted in the same piece that BERT inference is “really slow” by comparison.

Add query-family, language, freshness, source, and permission slices. Inspect answerability and evidence support when retrieval feeds a generator, rather than assuming ranking metrics capture the final product.

Start with twenty queries and retrieve top candidates from lexical and dense systems. Have reviewers identify irrelevant but plausible documents, duplicate positives, and unresolved judgments.

Add verified hard negatives to training. Then retest candidate recall and exact-identifier performance. Check whether the new model overfits one negative source or suppresses alternative relevant passages.

The review step is the cheapest one to skip and the most expensive to have skipped. The cost has been measured. RocketQA mined hard negatives from a retriever's own top-k in 2021, then checked what it had picked up: “Specifically, we manually examine the top-retrieved passages of 100 questions, that were not labeled as true positives. We find that about 70% of them are actually positives or highly relevant.” Training on those undenoised negatives took MS MARCO MRR@10 to 26.03. Plain in-batch negatives gave 32.39, and cross-batch negatives 33.32. The mining had made the model worse than not mining at all. The same negatives, filtered first with a cross-encoder, gave 36.38; with data augmentation, 37.02. The distance between 26.03 and 36.38 is a review pass over identical mined data.

Look at how DPR defines a hard negative before copying the recipe. Karpukhin and colleagues use passages “that have high BM25 scores given the question, but do not contain the answer string”. They report that “adding a single BM25 negative passage improves the result substantially while adding two does not help further”. A passage can answer a question without containing the answer string. That filter is an approximation with a known failure direction. It is the reason the human review step in this exercise is not optional.