Glance: The Generator Is Its Own Reranker

AI summary

Overview

Research area: Retrieval-augmented generation (RAG), passage reranking, and efficient LLM serving.

Technical level: Advanced. The paper assumes familiarity with cross-encoder rerankers, KV caching, LoRA adapters, and hybrid linear/full-attention architectures.

Scope: The paper introduces Glance, a method that lets a single frozen 9B generator score all candidate passages itself from one shared prefill, and reports retrieval results on two BEIR datasets (SciFact, NFCorpus), downstream QA on HotpotQA, and serving/memory measurements across candidate counts.

What This Paper Is About

Typical RAG systems serve two models: a reranker that encodes every query–passage pair separately, and a generator that then reads the winning passages. This duplicates model residency and passage processing, since the generator is going to read those passages anyway. Glance asks whether the generator can judge all candidates itself, using computation shared across passages, so that one resident backbone both reranks and answers.

Key Contributions

1. Method. Passage-specific relevance probes that branch from one shared passage prefill, read by the generator's own backbone through a small adapter (rank-16 LoRA plus a 256-dimensional pointer head), including on hybrid linear/full-attention backbones where a linear-attention layer carries a single recurrent state through the sequence.
2. Evidence. Retrieval results on two BEIR datasets (SciFact, NFCorpus), both zero-shot and after short MS MARCO training; downstream QA on HotpotQA under the same base generator; and a scaling study up to 200 candidates (43k tokens) per shared prefill.
3. Unified serving. Small decision adapters that share one backbone and are switched per request, with outputs identical to each model loaded alone, at approximately 0.1 GiB of additional memory per decision skill, plus an activated-adapter variant whose answer continues from the reranking cache.

Main Findings

• Trained reranking beats the cross-encoder baseline on both BEIR sets. After 39 minutes of MS MARCO fine-tuning, Glance reaches nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus, versus 0.721 and 0.326 for bge-reranker-v2-m3 (BM25 top-20; differences are not significance-tested).
• Zero-shot one-pass scoring is competitive but not better everywhere. With no retrieval training data, packed_all scores 0.715 on SciFact (vs. 0.729 for 20 separate decision calls to the same 9B model, and 0.721 for the cross-encoder) and 0.341 on NFCorpus (vs. 0.323 for separate calls and 0.326 for the cross-encoder).
• Shared prefill is much cheaper than sequential scoring, but the cross-encoder is still faster. Zero-shot packed_all took 614 ms (SciFact) and 670 ms (NFCorpus) per query on a shared RTX PRO 6000, against 2210 ms and 2485 ms for 20 separate 9B calls, while the cross-encoder took 31 ms and 60 ms. Trained packed_all latency on a B200 was 420 ms (SciFact) and 411 ms (NFCorpus).
• Better downstream answers under the same generator. On 300 HotpotQA distractor questions, Glance's top-3 context improves answer F1 from 0.684 (cross-encoder top-3) to 0.729 with the adapter off, a gain of 0.045 (paired bootstrap 95% CI [0.014, 0.076]). With the trained adapter left on for both contexts the gap is 0.065 (CI [0.038, 0.094]).
• Zero-shot Glance retrieves fewer gold paragraphs but answers better. Its R@2 is 0.762 versus 0.863 for the cross-encoder, yet F1 with the adapter off is 0.718 versus 0.683 — a gap the authors report as not significant (CI [−0.003, +0.074]) — and level with giving the generator all 10 paragraphs (0.716).
• Quality keeps rising with candidate count when the adapter is trained. On the first 100 SciFact queries with 100-word passages, the zero-shot adapter peaks at k=20 (0.662) and degrades to 0.584 at k=100. The trained adapter improves with k, reaching 0.682 at k=100 (21.6k tokens) against 0.637 for the cross-encoder, and the trained-long adapter reaches 0.697 at k=100 and 0.717 at k=200 (43.4k tokens) against 0.659 for the cross-encoder.
• Memory, not quality, is the binding constraint. Peak memory grows from 15.4 GiB at k=10 to 56.4 GiB at k=100 because each probe branch copies the shared state's attention cache; the zero-shot single call runs out of memory at k=200 under a 64 GiB cap, and the chunked path runs k=300 (65k tokens) in 23 GiB at about 22 s per query. Batching queries currently lowers throughput from 2.87 queries/s (B=1) to 1.57 (B=16) and runs out of memory at B=32.
• Mode switching is exact and cheap. Cycling base → general → rerank for 10 rounds left greedy chat text identical and probabilities bit-identical to each model loaded alone, with median switch times of 4.2–6.4 ms (base), 9.4 ms (general) and 8.8 ms (rerank), and 16.85 GiB resident memory unchanged after the loop.
• Specialising an adapter costs some general skill. On the 231 public items of JevBench, the general adapter scores 184/231 (79.7%) and the reranking adapter warm-started from it scores 175/231 (75.8%), a net loss of 9 items, 7 in the hard tier.
• The activated variant matches the control and enables cache reuse. Trained for 150 steps on 1,200 MS MARCO records, the activated adapter reaches 0.740 / 0.351 nDCG@10 on SciFact / NFCorpus versus 0.744 / 0.351 for a matched all-token control. On 100 HotpotQA questions, continuing the original cache gave EM/F1 of 0.61 / 0.750 versus 0.59 / 0.754 for re-prefilling the top-3, with lower median answer latency (138 ms vs. 181 ms).
• Scores depend mildly on passage order. Across BM25 order and 4 random orders on 100 SciFact queries, nDCG@10 ranged from 0.678 to 0.693, the top-ranked passage matched BM25 order in 91% of cases, and the median per-passage probability shifted by 10^-4.
• A vLLM prototype works but is not exact. Serving 4 chat answers plus 19 decision rows took 0.76 s versus 0.43 s for chats alone; probabilities differed from the Kev runtime by 0.006 on average (maximum 0.10) and 4 of 400 decisions flipped, with throughput of 1.54 requests/s versus 0.86.

Methodology in Plain English

The authors keep a single frozen 9B model — MiMo-V2.6-Distill-Qwen-9B, a hybrid that interleaves Gated DeltaNet linear-attention layers with full-attention layers in a 3:1 ratio — and add two small trainable pieces: a rank-16 LoRA adapter and a 256-dimensional pointer head.

To rerank, they put all candidate passages into one context and prefill it once. From the end of that shared context they run one short question per passage (about 30 tokens each), for example "Does Passage i contain information that answers or supports the query?" The pointer head compares the model's hidden state at a decision marker with the hidden state at each passage's marker and produces one relevance score per passage. Because every branch starts from a copy of the cached state rather than re-reading the passages, no passage is prefilled more than once. Two alternative packings are tested: one where the query is the shared state and each branch holds a passage, and a listwise version where a single question offers the passages as options.

The adapter is trained first on 4.4k general decision records for 20 minutes (no retrieval data), then optionally on 4,000 MS MARCO queries with BM25 hard negatives — one epoch, 500 optimizer steps, 55,598 labelled probes, 7.7M forward tokens, 39 minutes on one B200 with 21 GiB peak memory. No BEIR or HotpotQA data is used in training.

Two answering pipelines are compared. In the main one, the adapter runs on all tokens, the top-n passages are re-prefilled with the adapter off, and the model answers. In the activated variant, the passages are prefilled with the base weights and the adapter is switched on only for the probe tokens, so generation can continue the original cache, which still holds all candidates.

Evaluation covers SciFact (300 test queries), NFCorpus (323 test queries), HotpotQA distractor (first 300 validation questions, 10 paragraphs, 2 gold), and a scaling study on the first 100 SciFact queries. Baselines are BM25 order, bge-reranker-v2-m3, and the same 9B decision model scoring each passage in a separate sequential call. Hardware includes an RTX PRO 6000 (96 GB), a B200 and an H100, and the authors caution that latencies come from different, partly shared GPUs.

Why This Matters

The paper tests a simple but consequential idea for RAG engineering: if the generator must read the passages anyway, the reranker may not need to be a separate model at all. It shows a path to collapsing two served models into one resident backbone with roughly 0.1 GiB of extra memory per decision skill, while keeping the ability to switch between plain chat, general classification decisions and reranking per request without reloading weights. The results are explicitly framed as a proof of concept — the cross-encoder is still far faster per query, and the evaluation covers two retrieval datasets, one QA dataset and one backbone family — but the serving and memory accounting is the kind of detail that matters when deciding whether to deploy one model or two.

Real-world applications:

• Enterprise and internal search assistants that must fit a generator and a reranker into one GPU budget.
• Multi-tenant RAG services that need chat, classification and reranking modes from a single resident model with per-request switching.
• Long-document or many-passage pipelines (up to 200–300 candidates per shared prefill in these experiments) where re-encoding every pair is the bottleneck.
• Agentic or tool-using systems where the same backbone must make many small typed decisions between generation steps.

Industry relevance: The paper reports concrete memory (16.85 GiB resident for three modes) and latency numbers, a vLLM prototype, and released adapters under Apache-2.0, which makes it directly actionable for teams weighing dedicated rerankers against a unified backbone.

Future Directions

• Larger and broader benchmarks. The authors ask how the benefits extend across larger evaluation sets and other backbone families, given that current results cover hundreds of queries and one hybrid 9B model.
• Removing the per-branch cache copy. Memory currently scales with k × context length; letting all branches attend to one shared cache with a block mask would make memory linear in context length and could make a reranker over a 128k-token shared prefill feasible.
• Optimised serving. A production vLLM connector that exports only the readout positions and runs the pointer head inside the engine, plus a re-examination of the latency trade-off against small cross-encoders on an optimised path.
• Training for answer utility rather than relevance. The labels are passage relevance, not answer quality, which the authors note does not show the model has learned which evidence is useful for answering.

Target Audience

Machine learning engineers and applied researchers building RAG systems who are deciding how to serve reranking alongside generation; efficiency and inference-systems practitioners interested in shared-prefill and cache-reuse techniques; and NLP researchers working on LLM-based reranking, adapter composition, and hybrid linear-attention architectures. Readers without background in KV caching or LoRA will find the systems sections dense.

Authors’ abstract

Retrieval-augmented generation commonly serves separate models for reranking and answering, duplicating model residency and passage processing. We introduce Glance, which equips the generator with a lightweight decision adapter and pointer head to score candidate passages from a shared prefill. Short relevance probes branch from the cached context, producing one score per passage without separately prefilling each query-passage pair. A single resident 9B backbone supports generation, general decisions and reranking, with approximately 0.1 GiB of additional memory per decision skill. After 39 minutes of MS MARCO fine-tuning, Glance achieves nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus, compared with 0.721 and 0.326 for bge-reranker-v2-m3. On 300 HotpotQA questions, its selected context improves answer F1 by 0.045 over cross-encoder-selected context under the same base generator (paired bootstrap 95% CI [0.014, 0.076]). In a scaling study, reranking quality keeps rising up to 200 candidates (43k tokens) scored from one shared prefill, where it exceeds the cross-encoder by 0.058 nDCG@10, and a probe-only activated adapter lets generation continue from the original passage cache. These initial results demonstrate generator-based reranking with one resident backbone; evaluation currently covers two retrieval datasets, one QA dataset and one backbone family.