Context Windows, Position Effects, and Effective Recall

Accepting a long sequence is not the same as using every part of it equally well. Relevant evidence can be diluted by distractors, placed where retrieval is weak, or contradicted by later context. Context length is therefore an interface limit, not a guarantee of recall or reasoning.

The gap opens early. FLenQA, a benchmark built for a 2024 paper called "Same Task, More Tokens", holds the reasoning problem fixed and pads it out: three tasks, 100 base instances each, at 250, 500, 1000, 2000 and 3000 tokens. Length was the only thing that changed. Across GPT-4, GPT-3.5-Turbo, Gemini Pro, Mistral Medium and Mixtral 8x7B, average accuracy fell from 0.92 at the shortest length to 0.68 at 3000 tokens. A quarter of the score, gone, at a length far below any of those models' advertised maximum. The abstract puts it plainly: “Our findings show a notable degradation in LLMs’ reasoning performance at much shorter input lengths than their technical maximum.”

Effective context depends on model training, position handling, prompt structure, and the application's selection policy. None of those is published on a specification sheet.

Increasing the outer limit does not remove the inner bottlenecks.

The architectural limit is the one most often treated as a label rather than a design decision. How a model represents position decides how far it can run beyond the lengths it was trained on. One method, ALiBi, adds no positional embeddings at all. It penalises query-key attention scores in proportion to distance. It arrived at ICLR 2022 under the title "Train Short, Test Long", and the abstract reports the result: “We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory.” Doubling the usable input was a property of how position was represented. Not of how much data the model had seen.

These failures can occur long before the formal token limit is reached, and the price of the second one has been measured precisely.

GSM-IC takes arithmetic word problems from GSM8K and adds one irrelevant sentence to each: 58,052 examples in all. A single extra sentence, and: “The drop on macro accuracy is especially large, showing that fewer than 30% of the base problems are consistently solved after adding distractors.” That is from the ICML 2023 paper the benchmark was built for, "Large Language Models Can Be Easily Distracted by Irrelevant Context". The best macro accuracy among all the prompting techniques its authors investigated was 45%. The distracting sentence broke nothing in the model and removed nothing from the window. It was simply there.

The correct choice depends on whether missing information or excess information limits the task.

The third option — letting the conversation itself accumulate as state — is the one that looks free and is not. A 2025 study, "LLMs Get Lost In Multi-Turn Conversation", simulated more than 200,000 conversations. Each task was delivered either as one fully specified instruction or spread across turns. Nothing was removed in the multi-turn version. The same information simply arrived in pieces. “Our experiments confirm that all the top open- and closed-weight LLMs we test exhibit significantly lower performance in multi-turn conversations than single-turn, with an average drop of 39% across six generation tasks.” Its authors decompose that 39% into a small loss of aptitude and a large rise in unreliability. The model does not get much less able. It gets much less predictable. Compression and restatement are not merely a cost control. They are a correction to the shape history arrives in.

Vary one property at a time so failures are interpretable.

NoLiMa is a worked instance of the first four steps, and of what happens when one design flaw is removed. It rebuilds the needle-in-a-haystack test so that the question and the needle share minimal lexical overlap. The model has to associate rather than string-match. Its authors swept length across 13 models, all claiming at least 128K tokens, and reported at ICML 2025: “At 32K, for instance, 11 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%.”

The lesson for your own harness is in the contrast with the 99%-at-1M headline. The same family of test gives a near-perfect score, or halves 11 models out of 13, depending on whether the needle repeats the question's wording. If your planted evidence shares vocabulary with your query, you are measuring your string matcher.

A larger window can reduce truncation and support richer tasks. It can also cost more, crowd the evidence with material that distracts, keep stale state around, and hide a privacy exposure. The measurements in this lesson all point the same way: accuracy falling from 0.92 to 0.68 at 3000 tokens, fewer than 30% of problems solved consistently after one irrelevant sentence, 11 of 13 models halving at 32K, a middle position scoring below closed book, half of 17 models missing the length on their own label.

The next lesson focuses on context engineering: deciding what enters the window, how it is ordered, and what should be compressed, retrieved, or excluded.