Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

AI summary

Overview

Research area: Natural Language Processing — long-context language models, retrieval, and attention analysis.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with transformer attention and softmax normalization to follow the central argument.

Scope: A single-author empirical study arguing that retrieval failures in long contexts are driven by content competition for attention rather than by token count, tested on five open models of 0.5B to 3B parameters across 126,473 trials.

What This Paper Is About

Language models are widely reported to degrade as their inputs grow, a phenomenon called "context rot," and the usual explanation is length itself. This paper tests the opposite hypothesis: that for retrieval, adding neutral tokens is nearly free, and what actually breaks the model is other statements competing with the evidence for the same query. The goal is to identify a measurable internal quantity — the attention the model leaves on the evidence — that predicts when retrieval fails.

Key Contributions

1. Separating length from content. The authors build four matched conditions (short, neutral, competing, masked) that hold total length and evidence position fixed while changing only what surrounds the evidence, and show that neutral text costs almost nothing while template-matching competitor statements destroy accuracy.

2. Attention mass as a trial-level predictor of loss. Across 126,473 trials, log(1/a*), where a* is the attention sent from the final query token to the evidence tokens by eight selected (layer, head) pairs, explains 35 to 58% of the variance in loss, against 2 to 8% for log length.

3. A failed control in prior work. The authors replicate the masked-filler protocol of Du et al. on MMLU (11 to 18 points lost at 30K masked tokens on Llama-3.2-3B and Qwen2.5-7B), then show that in retrieval the same masking costs 46 accuracy points at 32K tokens while identical filler left visible costs none.

4. An open probe and dataset. The harness, attention probe, and all trial-level measurements are released (code at github.com/yliusci/length-is-cheap, data at huggingface.co/datasets/yliusci/length-is-cheap).

Main Findings

• Neutral length is cheap. With visible neutral filler (random numbers, random words, or one repeated sentence), accuracy stays between 0.97 and 1.00 on four of five models up to 4K tokens, and up to 32K on Qwen2.5-1.5B. On Qwen2.5-1.5B the median loss stays at or below 0.006 nats at every length from 8K to 32K for all three neutral fillers.

• Competing statements are not. Filling the same positions with statements in the evidence template drives accuracy to 0.11 to 0.21 at 8K to 32K tokens. On Qwen2.5-1.5B the median loss reaches 2.82, 3.94, 4.03 and 4.76 nats at 8K, 16K, 24K and 32K; on Qwen2.5-0.5B it reaches 1.79 to 4.07 nats with accuracy 0.17 to 0.40.

• Loss concentrates on same-key conflicts. Contexts without any statement sharing the queried key lose 0.02 to 0.34 nats; contexts with at least one lose 1.38 to 3.24 nats. Accuracy is 0.97 to 1.00 without conflict and 0.25 to 0.52 with one. At fixed length on Qwen2.5-1.5B at 2,048 tokens, loss is 0.03, 1.37, 2.24, 2.48, 2.71, 3.26 and 3.45 nats for 0 to 6 or more conflicts. A single statement giving the queried key a rival value multiplies the loss by 9 to 110.

• Format alone barely matters. Explicit distractors that share the format but never the key explain at most 0.3% of the variance in loss (R² ≤ 0.003).

• Attention predicts, length does not. R²(log(1/a*)) ranges from 0.35 to 0.58 across the five models, against 0.02 to 0.08 for log L. Attention on the evidence collapses from 0.51 to 0.77 in the short context to 0.17 to 0.31 with competing filler. From 8K to 32K tokens, R²(log(1/a*)) is 0.32 on Qwen2.5-1.5B and 0.33 on Llama-3.2-3B, against 0.02 and 0.03 for log L; on SmolLM2-1.7B up to its 8K window it is 0.45 against 0.18.

• Attention subsumes conflict count. Once log(1/a*) is known, the number of conflicts raises R² by at most 0.04. Within conflict-free contexts, where losses are small, log(1/a*) explains only 2 to 10% of the variance.

• The relationship is thresholded. On Qwen2.5-1.5B the loss stays near zero until a* falls to about 0.35, then rises steeply. Slopes of loss on log(1/a*) exceed 1 on every model (1.40, 2.22, 1.68, 3.97, 2.75 for Qwen2.5-0.5B, Qwen2.5-1.5B, SmolLM2-1.7B, OLMo-2-1B and Llama-3.2-3B), and the authors explicitly decline to report a universal law.

• Masking is not a length baseline. On Qwen2.5-1.5B at 32K tokens, the median cost of neutral numeric text is 0.004 nats, of masking it 1.20 nats, and of competing statements 4.76 nats. Across the three neutral fillers at 32K, accuracy is 0.96 to 0.99 when visible and 0.48 to 0.54 when masked. The effect is model-dependent: on Qwen2.5-0.5B masking costs 0.1 to 0.4 nats.

• MMLU replication succeeds but is not a clean length control. MMLU accuracy with masked filler at 0, 3,750, 7,500, 15,000 and 30,000 tokens is 51.7, 44.7, 42.4, 39.3 and 40.5 for Llama-3.2-3B and 65.8, 62.6, 54.5, 49.4 and 48.1 for Qwen2.5-7B, with paired McNemar tests at p ≤ 0.007. At 4,096 tokens on Qwen2.5-1.5B, MMLU accuracy is 59.1% with no filler, 57.5% with visible neutral filler and 53.9% with masked filler, while visible whitespace padding drops the same model to 36.0%.

• One model is an exception. OLMo-2-1B loses 0.57 nats with neutral filler at 2,048 tokens (accuracy 0.91).

Methodology in Plain English

The design rests on one rule the author calls the visible-control protocol: every comparison holds total length and the position of the evidence fixed, and changes only what surrounds the evidence. This requires only forward passes, so it applies to any open-weight model.

The task is key-value retrieval. Each context hides one evidence statement of the form "The secret code of the river is 4821," where the key is a common noun and the value a four-digit number. The context ends with a fixed query, and the authors score the log-probability the model assigns to the correct value.

For each trial, four contexts are built with identical evidence, query and length. In the short condition nothing surrounds the evidence. In neutral the filler is random numbers, random words, or one repeated sentence. In competing the filler consists of statements in the same template, with keys drawn uniformly from 51 nouns — so some of them collide with the queried key and assign it a different value. In masked the filler is removed from attention by a 2D mask while position indices continue counting through it.

The central measurement is a*: for each configuration, the eight (layer, head) pairs that place the most attention on the evidence tokens in the short context are selected, and a* is the attention those heads send from the final query token to the evidence tokens. Predictors are compared by the R² of univariate least-squares fits on a*, log(1/a*) and log L.

Models are Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, SmolLM2-1.7B-Instruct, OLMo-2-0425-1B-Instruct and base Llama-3.2-3B, at lengths of 512, 1,024 and 2,048 tokens for all models and up to 32,768 within each model's window. Runs use float16 weights and PyTorch SDPA attention on one Apple workstation (64 GB unified memory) and one RTX 4090 laptop GPU (16 GB). The replication uses 1,000 MMLU questions on Llama-3.2-3B and Qwen2.5-7B-Instruct with 3,750, 7,500, 15,000 or 30,000 masked tokens, implemented by deleting the filler and advancing position indices by the same amount — verified to match explicit attention masking on Llama.

Why This Matters

Impact on research. The paper challenges the interpretation of masked-filler experiments, which have been the main evidence that length alone degrades performance. It argues that a context of masked tokens at preserved positions presents the model with long positional gaps absent from pretraining, so its cost should not be read as the cost of length. It also connects the previously separate literatures on knowledge conflicts and attention dilution by measuring behavior, content and attention in the same trials, and it supplies an interpretable diagnostic that requires only a forward pass.

Real-world applications:

• Retrieval-augmented generation: deduplicating and resolving conflicting statements before they reach the model should matter more than shortening the retrieved context.
• Agent memory: deciding what an agent retains across sessions is a curation problem, not a truncation problem.
• Long-document pipelines: summarization and question answering over lengthy documents can tolerate thousands of neutral tokens but not a handful of near-duplicate or contradictory passages.
• Monitoring and diagnostics: a* can be computed at inference time as an early warning that a model is about to lose the evidence, without access to labels.

Industry relevance. The finding reverses a common engineering priority. Systems that aggressively compress or truncate context may be trading away useful material while leaving the actual culprit — competing statements — in place. Because a* needs only a forward pass, it is deployable as a cheap runtime signal on any open-weight model.

Future Directions

• Causal intervention on attention. The authors note that a* is measured in the same forward pass as the loss and that they ran no intervention on it, so the evidence is predictive rather than causal. Directly manipulating attention on the evidence would test the claim.
• Measuring format-only competitors above 2K tokens. In the competing filler, the number of conflicts grows with length and every context of 4,096 tokens or more contains at least one, so the effect of competitors that share only format is unmeasured at long lengths. The conflict analysis was also not preregistered.
• Testing whether attention-sharpening methods help. The authors predict that methods which sharpen attention at long range help most when they keep a* above the failure threshold, a prediction the paper does not test.
• Extending beyond synthetic retrieval and small models. All results come from one synthetic retrieval task and models of at most 3B parameters, except the 7B model in the MMLU replication, which has no visible-filler arm beyond 4K tokens.

Target Audience

Researchers and engineers working on long-context language models, retrieval-augmented generation, and agent memory; practitioners who design context-assembly pipelines and need to decide whether to compress or curate; and interpretability researchers interested in attention-based diagnostics. It is most useful to readers who already understand transformer attention and want an empirical, trial-level account of why long contexts fail.

Authors’ abstract

Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110, and every additional rival adds to it. Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials it explains up to 58% of the variance in loss; log length explains at most 8%. The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval: masking costs 45 accuracy points at 32K tokens, while the same filler left visible costs none. In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements.