Equal Ranking Quality, Different Decisions: Measuring and Reducing Order Dependence in LLM Scorers

AI summary

Overview

• Research area: Natural Language Processing, specifically LLM-based document/response scoring, passage reranking, retrieval evaluation, and preference-model training data.
• Technical level: Advanced. The paper assumes familiarity with nDCG, Kendall's tau, LoRA fine-tuning, and distillation; the framing, however, is unusually decision-oriented and readable.
• Scope: The paper measures how much a scored decision changes when the same candidates are reordered in a shared prompt, and introduces a training objective, order-consistency SFT (OC-SFT), that reduces that change across passage reranking, multi-document QA, and response ranking.

What This Paper Is About

When an LLM scores several candidate documents or responses together in one prompt, each candidate still receives its own score, but that score depends on the order in which the candidates appear. Systems are selected on ranking quality (such as nDCG@10), yet downstream consumers act on the scores themselves: a threshold admits documents, a reader answers from the top few, and a preference model picks a chosen/rejected pair. The paper shows that two scorers with nearly identical ranking quality can produce very different decisions under reordering, and it proposes a training penalty that makes those decisions more reproducible without hurting ranking quality.

Key Contributions

1. Measuring decisions rather than rankings. The authors show that equal ranking quality does not imply equal decisions: five trained scorers within 0.010 nDCG@10 on passage reranking retain sets that overlap by only 0.66–0.84 when candidates are reshuffled. They argue that comparisons of such scorers should report what a threshold retains and what a reader answers, not ranking quality alone, and that a stability metric's permutations are sufficient to run that measurement.
2. Showing that prompt-construction fixes do not reach the decisions. Round-robin partitioning gains 0.052 nDCG@10 but changes none of the three consumer decisions measurably, and logit calibration (CapCal) improves neither quality nor retained-set overlap measurably. Neither fix reaches any of the three consumers.
3. Introducing order-consistency SFT (OC-SFT). OC-SFT penalizes disagreement between a candidate's scores across orderings in the weights themselves, holding ranking quality while leading every decision-stability measure among trained scorers on all three tasks (reader-answer flip rate 0.125 versus 0.149–0.164 for three other order-aware objectives).
4. Comparing OC-SFT with order-averaged distillation and inference-time averaging. OC-SFT is more stable on 12 base models than order-averaged distillation (which trains on labels averaged across permutations), and one OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. Code is released at github.com/thomsonreuters/presentation-dependence.

Main Findings

• Ranking quality does not predict decision stability. Five trained variants in Table 1 differ by at most 0.010 nDCG@10 on passage reranking, less than a permutation moves a single query's score under any one of them, yet retained-set overlap ranges from 0.656 (single-order distillation) to 0.835 (OC-SFT), a span of 0.179. Among seven single-permutation systems within 0.022 nDCG@10, different systems take the best quality and the best retained-set overlap on 11 of the 18 reranking collections.
• One query can keep its nDCG@10 and change its answer. In Figure 1a, off-the-shelf Qwen3-4B on Climate-FEVER over the same 100 candidates and one query gives nDCG@10 of 0.6131 under both permutation A and permutation B, retains 5 of 100 documents in both cases, but the reader returns SUPPORTED under one ordering and NOT ENOUGH INFO under the other.
• Order changes preference training labels too. Off the shelf, the order decides whether a preference model's top choice is right on 65.4% of queries across ten random permutations.
• Prompt-time fixes do not help the decisions. Round-robin partitioning recovers 0.052 nDCG@10 on reranking without extra permutations, but retained-set overlap, verdict flip on either claim-verification collection, and pair flip do not change measurably. CapCal leaves instability near the off-the-shelf level, and rescaling across windows cannot correct the partition effect because the windows do not differ by an offset. No in-format placeholder value for the answer skeleton lowers order instability distinguishably.
• OC-SFT reduces instability sharply. On passage reranking, order instability (τ-PSI) falls from 0.209 under single-order distillation to 0.083 under OC-SFT, and the retained set, the reader's output, and the selected pair all become more reproducible. OC-SFT also narrows per-query score movement (spread 0.046, against 0.079 for single-order distillation, 0.064 for order-averaged distillation, 0.062 for DebiasFirst, 0.061 for permutation augmentation, and 0.135 off the shelf).
• The gain holds across models and perturbations. OC-SFT stays below order-averaged distillation in τ-PSI on all 12 bases, including the sparse Mixture-of-Experts control; quality and retained-set overlap separate the same way across 198 base-collection cells over 11 dense bases (pooled quality change +0.020, pooled retained-set overlap change +0.206). Under pool replacement, an untrained perturbation, instability is 0.031 against single-order distillation's 0.099.
• A larger quality number can hide less reproducibility. GPT-5.4 leads every trained variant on nDCG@10 (0.468) while reproducing less of its retained set (0.707) than OC-SFT (0.835). jina-reranker-v3 leads OC-SFT on retained-set F1 while trailing on nDCG@10 (0.447 against 0.459), and its retained sets overlap less (0.667 against 0.835).
• The penalty, not the extra views, does the work. Permutation augmentation, which trains on the same shuffled views but omits the explicit consistency penalty, and DebiasFirst each close about two thirds of the τ-PSI gap. A generic EMA mean-teacher closes 71% of the gap but costs the reader 0.023 exact match.
• Training beats inference-time averaging at a tenth of the inference. OC-SFT at one permutation (K=1) beats batched self-consistency at K=10 on two of the three consumers; BSC reaches OC-SFT's K=1 level only on response ranking, near K=8.
• Training and averaging compose. Averaging the trained scorer keeps improving it, so training does not substitute for averaging. Order-averaged distillation pays in labels instead, needing T=10 teacher passes against OC-SFT's one, and is less stable on 30K labeled queries than OC-SFT on 1K.
• Shared context helps unevenly. Serving a checkpoint at its training width rather than at B=1 gains 0.043 nDCG@10 on multi-document QA; on passage relevance at B=20 the width effect is indistinguishable from zero; on response ranking the gain is largest on the PPE collections while RM-Bench loses. Batched-teacher labels beat pointwise ones at both training widths, by 0.029 nDCG@10 at B_train=1 and 0.010 at B_train=20, both evaluated at B=1, and batching holds that quality at 45% lower per-query latency.

Methodology in Plain English

The researchers start by defining what can vary about a prompt while the query and the candidate pool stay fixed: the order of candidates, the rule that splits the pool into windows, the markers that label slots, and the wording of the answer skeleton. They call one setting of all four a "presentation." They then define order instability with τ-PSI, a rank-level adaptation of an existing position-sensitivity index: each instance is scored under M = 10 random permutations, pairwise Kendall correlations between the induced rankings are averaged, and the result is scaled so that 0 means every order gives the same ranking, 0.5 means rankings are uncorrelated, and 1 means reversed.

To score candidates without generating text, they append a fixed answer skeleton with one placeholder grade token per candidate, requesting an integer grade in {0, 1, 2, 3}. A single forward pass yields the model's distribution over the four grade tokens at each position; the expected grade, normalized to [0, 1], is the score. One prompt therefore yields all B scores, and no decoding temperature enters.

They decompose the scorer output into an order-marginal term (the score a permutation-invariant scorer would emit) and an order residual (what remains and varies with order). Their proposed objective, OC-SFT, keeps the ordinary distillation target on one view and adds a penalty that pulls each of N = 2 shuffled views toward their mean. The penalty is proportional in expectation to the squared order residual, so it directly attacks order dependence in the student's own weights, while leaving the order-marginal unconstrained. They compare this against single-order distillation, order-averaged distillation (which bakes the averaging into the labels), batched self-consistency (which averages at serving time), permutation augmentation, DebiasFirst, and calibration methods.

Evaluation covers three tasks. Passage reranking is primary: trained on MS MARCO at B = 20 with labels from a teacher grading the top-100 candidates of roughly 30K queries, evaluated on 18 collections (five TREC Deep Learning collections DL19–DL23, 11 out-domain BEIR collections, and two internal legal collections, with BEIR and legal evaluated zero-shot). Multi-document QA uses the HotpotQA distractor setting with ten passages per question, trained on 30K questions and transferred zero-shot to 2WikiMultiHopQA and MuSiQue. Response ranking trains on roughly 30.7K UltraFeedback prompts with four candidates each and evaluates on five collections. Qwen3-4B is the primary base, with 11 dense bases from 1.7B to 32B across Qwen3, Gemma 4, and Granite 4.1, plus two controls. Decisions are measured with frozen per-collection F1-tuned cutoffs fitted on half a collection's queries and evaluated on the other half, and with two frozen readers consuming the top 5, Granite-4.1-8B throughout and Qwen3-4B as a second reader.

Why This Matters

Impact on research. The paper argues that ranking quality alone is an insufficient basis for comparing LLM scorers, because it reads only the discounted order of the top ten while a threshold reads absolute scores. It shifts the evaluation question from "how does the ranking move" to "what do the consumers of that score decide," and shows a stability metric's permutations are enough to run that measurement. It also demonstrates that prompt-side interventions, including ones that improve ranking quality such as round-robin partitioning, do not reach downstream decisions.

Real-world applications:

• Legal and high-stakes search review queues. The paper explicitly frames reordering-induced score variation as a reproducibility concern in legal search, and evaluates on two proprietary legal collections (Legal-A and Legal-B) from a commercial legal search provider with attorney relevance judgments.
• Retrieval-augmented generation. A reader answers from the highest-scoring few documents, so unstable scores change what gets retrieved into the generation context and what the reader outputs. The paper measures this with verdict flip on claim verification and answer flip on QA.
• Preference data and reward modeling. A preference model's scores determine which response becomes the chosen/rejected pair in training data. Off the shelf, order decides whether the model's top choice is right on 65.4% of queries, so an unstable scorer introduces noise directly into training labels.
• Threshold-based triage and routing. Any system that admits documents above a cutoff into a queue inherits the cutoff's sensitivity to candidate order; retained-set overlap under reshuffling ranged from 0.656 to 0.835 among quality-matched scorers.

Industry relevance. The reported cost profile matters operationally: OC-SFT needs one teacher pass and serves one permutation, whereas order-averaged distillation needs T = 10 teacher passes and batched self-consistency multiplies serving cost by the number of permutations. At B = 20, batching holds quality at 45% lower per-query latency. The work also comes with open code from a commercial research lab and reports results across three model families and 12 bases, including a sparse Mixture-of-Experts control.

Future Directions

• Reconciling training with averaging. The paper reports that averaging the trained scorer keeps improving reproducibility, so training and averaging compose rather than substitute. How far the two together can go, and whether the residual can be removed rather than attenuated, remains open.
• Extending beyond order. The decomposition runs over permutations of a fixed candidate set, so replacing a candidate changes the instance rather than the permutation and falls outside it. Pool composition and prompt wording are separate channels the paper holds fixed or varies only partially; the authors note that no skeleton setting is neutral and that position-bias measurements rarely report sensitivity to that choice.
• Choosing a serving design. The paper finds that serving width and the choice between pointwise and batched scoring depend on the serving budget and the task, and that batched-teacher labels help either way. A principled rule for selecting width per deployment, and an explanation for why a pointwise student degrades at 20 candidates while a batched checkpoint holds quality across 1–30, is not settled here.
• Testing whether the attenuation transfers to other scorer families and consumers. The comparison includes a frontier prompted scorer (GPT-5.4), a purpose-built reranker (jina-reranker-v3), and additional systems evaluated at native geometries (RankZephyr, mxbai-rerank-large-v2), but nearly half of GPT-5.4's candidate pairs tie and the authors omit its τ-PSI, leaving open how decision stability should be measured for scorers with heavy tie rates.

Target Audience

Researchers and practitioners working on LLM-based reranking, retrieval evaluation, reward modeling, and LLM judges, particularly those who select scorers on ranking metrics and then deploy them behind thresholds, readers, or preference-data pipelines. It is also relevant to evaluation-methodology researchers interested in stability and reproducibility metrics, and to engineering teams deciding between inference-time ensembling and one-time training to reduce order sensitivity. Readers should be comfortable with ranking metrics and supervised fine-tuning; the decision-level framing itself is accessible without deep mathematical background.

Authors’ abstract

In passage reranking, response ranking and multi-document question answering, LLMs can score several candidate documents or responses together in one prompt, each still receiving its own score. Such scorers are selected on ranking quality, but their scores determine a decision: what a score threshold retains, a reader answers, or which chosen/rejected pair enters preference training. Because the candidates share that prompt, reordering them changes their scores. The same query over the same candidates should still yield the same decision. However, equal ranking quality does not imply equal decisions: on passage reranking, five trained scorers within 0.010 nDCG@10 retain sets that overlap by only 0.66-0.84 when reordered. No prompt-time change we test resolves that dependence: the only one that improves ranking quality does not measurably improve decision stability. We introduce order-consistency SFT (OC-SFT), which attenuates it in the weights by penalizing disagreement between a candidate's scores across orderings. It holds ranking quality and leads every decision-stability measure among trained scorers on all three tasks. It is also more stable on 12 base models than order-averaged distillation, which trains on labels averaged across permutations. One OC-SFT permutation retains sets that overlap more than ten averaged off-the-shelf permutations. A comparison of such scorers should therefore report what a threshold retains and a reader answers, not ranking quality alone. Code is available at https://github.com/thomsonreuters/presentation-dependence.