Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

AI summary

Overview

Research area: Natural Language Processing — long-context question answering, retrieval-augmented generation, and the evaluation of sequence-mixer architectures (Transformer, state-space, and hybrid Mamba models).

Technical level: Intermediate. The experimental design is intuitive (slide one document through a prompt), but the paper relies on paired bootstrap inference, Holm correction, and linear probing, so some familiarity with LLM evaluation helps.

Scope: An exploratory, paired-intervention study of how answer accuracy changes as a single gold document moves across ten positions in a fixed prompt, compared across Transformer, state-space, and hybrid Mamba checkpoints.

What This Paper Is About

Retrieval can put the right document into a prompt without guaranteeing the model will actually use it. Prior work documented a "lost in the middle" effect for Transformers, where evidence near the edges of a prompt is used better than evidence in the middle, but most of that work examined only Transformer models. This paper asks whether that position curve is the same for different sequence-mixer families — dense-attention Transformers versus recurrent state-space models like Mamba — by holding the question, the nine distractors, and the token budget fixed while moving only the gold document.

Key Contributions

1. A paired position-bias comparison across mixer families. The authors apply one identical intervention — moving a single gold document through ten slots among nine fixed distractors — to one Transformer and two state-space checkpoints at similar parameter counts on a shared corpus (800 exploratory questions each).

2. A clean negative-leaning result at the family level. The Transformer shows a positive primacy edge (+0.052, 95% CI [0.032, 0.072]) while the two state-space variants show −0.001 ([−0.016, 0.014]) and −0.018 ([−0.034, −0.003]); all three show positive recency. Paired state-space-minus-Transformer primacy interactions are −0.053 and −0.070, both Holm-corrected at p < 10⁻⁴.

3. A matched pure-versus-hybrid Mamba contrast that stays honest about uncertainty. Using checkpoints that share training data, tokenizer, scale, depth, and positional setup, with the hybrid adding a minority of attention layers, the hybrid shows a larger within-model primacy edge, but the paired hybrid-minus-pure difference (+0.019, CI [−0.006, 0.044], Holm p = 0.144) does not survive correction.

4. A set of robustness probes showing the curve is not an architecture-only signature. A corpus swap, a synthetic retrieval task, prompt-template variants, a linear position probe, and an attention-concentration analysis all appear as separate phases, with the paper explicitly framing the internal measurements as hypothesis-generating rather than mechanistic.

Main Findings

• Transformers show primacy; the tested state-space models do not. On the shared-corpus comparison, the Transformer's primacy edge is +0.052 [0.032, 0.072], versus −0.001 [−0.016, 0.014] for state-space v1 and −0.018 [−0.034, −0.003] for state-space v2. Both families still favor evidence near the end.

• Both families show recency. Recency edges are +0.075 [0.054, 0.097] for the Transformer, +0.077 [0.058, 0.097] for state-space v1, and +0.105 [0.086, 0.124] for state-space v2. The shared end advantage is the one pattern that holds across families.

• The family gap only appears once models can answer the task. Across five approximate size pairs, the Transformer-minus-state-space primacy gap is near zero at the two smallest scales and is +0.062, +0.069, and +0.053 in the three larger pairs. The paper calls this a "descriptive family-and-capability trend, not a causal scale threshold"; the smallest Transformer ceiling is only 0.009.

• The matched hybrid comparison is directional but inconclusive. Pure state-space primacy is +0.008 [−0.009, 0.024]; hybrid primacy is +0.027 [0.008, 0.047]. The paired hybrid-minus-pure primacy effect is +0.019 [−0.006, 0.044] with Holm p = 0.144, and the paired recency effect is −0.028 [−0.055, −0.001] with Holm p = 0.089. Neither survives correction, and the authors stress that different within-model significance levels do not establish a between-model difference.

• Synthetic retrieval reproduces Transformer primacy but hits a state-space ceiling. On 50 RULER single-needle examples at 2K tokens, Transformer primacy is +0.12 [0.05, 0.20], while both state-space models score 1.0 at every tested depth, preventing a family comparison on that task.

• A corpus swap changes accuracy without changing curve shape. Holding the state-space architecture fixed, the Pile-minus-SlimPajama primacy effect is −0.007 [−0.030, 0.016] and the recency effect is −0.016 [−0.044, 0.012] — no detectable edge-shape change.

• Prompt layout can create or nearly erase primacy. On 200 exploratory questions, repeating the question before and after the documents moves state-space primacy from 0.000 to +0.103. Placing the question only before the documents yields primacy of +0.020 and cuts recency from +0.070 to +0.010. An alternative template reduces Transformer primacy from +0.070 to +0.005.

• Position stays decodable across architectures even when response curves differ. A balanced linear probe over 1,200 states predicts edge-versus-middle position above a shuffled-label control in both families, suggesting position information remains available even when it is not used equally.

• Attention concentration near initial tokens covaries with primacy in the tested Transformers. The paper explicitly notes this is correlational, that scale confounds the association, and that no valid sink-blocking intervention was completed.

• Harness controls behave as expected in one setting only. On the 200-question exploratory Transformer subset, the released key-value task produces a +0.38 edge-versus-middle effect, sham-gold primacy is +0.005 [−0.0325, 0.0425], and permuting distractor order changes mean accuracy by at most 0.023. The authors state these controls do not cover the matched 8B checkpoints.

• Additional systems all show positive primacy, but the comparison is not diagnostic. Three further Transformer and hybrid systems show primacy edges of +0.076, +0.041, and +0.067, differing in architecture, training data, tokenizer, token count, depth, and positional encoding — establishing prevalence, not mechanism.

Methodology in Plain English

The researchers took the released Lost-in-the-Middle ten-document question-answering setup built on Natural Questions. Each question comes with one gold document and nine distractors. For a given question they build ten prompts that keep the question, the distractors, and the total token budget identical, and move only the gold document — one prompt for each of the ten slots. They also build a no-gold prompt to measure the guessing floor and a gold-only prompt to measure whether the model can answer at all.

For each system they then compute two numbers: primacy, the accuracy at the start minus the accuracy in the middle, and recency, the accuracy at the end minus the accuracy in the middle. The unit of analysis is the question, not the individual generation, so complete ten-position bundles are resampled together across 10,000 paired bootstrap samples to get percentile 95% confidence intervals, with Holm correction applied within each family of comparisons. Generation is greedy, and the harness records prompts, outputs, failures, token counts, model revisions, and software settings.

Because the headline comparison could be confounded by depth, positional encoding, state size, and checkpoint family, they add a tighter contrast: pure and hybrid 8B Mamba checkpoints that share training data, tokenizer, scale, depth, and positional setup, where the hybrid simply adds a minority of attention layers.

They then run a series of scope checks — a corpus swap within one architecture, a synthetic retrieval task, prompt-template variants — and internal measurements: a linear probe for position decodability and an analysis of attention concentration near initial tokens. All ten-document results come from an 800-question exploratory allocation (dataset snapshot: 2,655 questions, with 800 for exploration and 1,855 held for confirmation); the confirmatory split remains unopened. The primary sweep contains 9,600 generations per system, with no reported generation failure or scored null, and the whole project used at least approximately 50 single-GPU hours.

Why This Matters

Impact on research. The paper's central claim is that available context length and effective evidence use are different properties, and that the widely cited middle-drop curve is not one universal shape. It argues that long-context evaluations should report full position curves, hold distractors fixed, use paired question-level inference, and test prompt sensitivity rather than reporting a single aggregate accuracy score. The explicit separation of an exploratory result from an unopened confirmatory split is also a methodological signal, as is the paper's refusal to name the sequence mixer as the sole cause.

Real-world applications:

• Retrieval-augmented generation pipelines — if evidence use depends on where the retriever places a passage, ranking decisions and prompt packing become accuracy decisions, not just formatting ones.
• Long-document and contract review tools — a fact buried in the middle of a 200-page filing may be systematically underused, and template layout could compensate.
• Agentic and multi-step systems that accumulate context — where retrieved material naturally drifts toward the middle as a conversation or tool trace grows.
• Model selection and procurement — teams choosing between attention-based and state-space/hybrid backbones now have evidence that equal aggregate benchmark scores can hide different failure profiles under different prompt layouts.

Industry relevance. State-space and hybrid architectures are attractive because of inference cost and throughput at long sequence lengths, and vendors report long-context benchmark numbers. This paper shows that a single aggregate score can mask position-dependent behavior, and that a prompt template change can shift the beginning advantage by more than the architecture choice does. For anyone deploying retrieval-augmented systems at scale, that is a direct argument for position-sweep testing before committing to a backbone or a prompt format.

Future Directions

• Train matched Transformer, state-space, and hybrid systems under one data and positional setup. The authors call for this explicitly, since the current family comparisons retain differences in depth, positional encoding, state size, and checkpoint construction.

• Run preregistered evaluation on the held-out 1,855 confirmatory questions. Every ten-document result reported here is exploratory; the confirmatory split has not been evaluated.

• Test targeted interventions on how evidence is retained and used. A valid sink-blocking intervention was not completed, and the linear probe does not show that answer content is stored or used — so the mechanism behind the family difference remains open.

• Extend paired position curves to longer contexts, retrieval-augmented generation, and multi-step tasks. This would test whether the observed patterns predict downstream reliability rather than only characterizing the ten-document setting.

Target Audience

Researchers and engineers working on long-context evaluation, retrieval-augmented generation, and efficient sequence architectures will get the most from this paper — particularly those comparing attention-based backbones against state-space or hybrid Mamba alternatives and needing to know whether a single benchmark score is a safe basis for choosing between them. It is also useful for practitioners who design prompt templates for multi-document QA, and for methodologists interested in how to structure exploratory versus confirmatory long-context evaluation with paired question-level inference.

Authors’ abstract

Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.