Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

AI summary

Overview

Research area: Natural Language Processing — long-context large language model evaluation, with connections to agentic AI, retrieval, and hallucination/safety alignment.

Technical level: Intermediate. The paper is readable without deep architectural knowledge, but assumes familiarity with concepts such as context windows, needle-in-a-haystack benchmarks, prompting conditions, and hallucination/refusal behavior.

Scope (one sentence): The paper introduces a model-agnostic extension of the needle-in-a-haystack benchmark that separates literal extraction, logical inference, and faithfulness, and uses it to measure how fact placement, nine corpus-level fact distributions, and anti-hallucination ("Don't Make It Up") prompts affect four production-scale LLMs.

What This Paper Is About

Long-context LLMs are increasingly used as an agent's working memory, with documents or tool outputs appended directly into a large prompt instead of being routed through external retrieval pipelines. Standard single-fact needle-in-a-haystack tests may overstate how well these models actually use that memory, because real evidence is usually scattered across a document rather than concentrated in one place, and because production systems often add strict anti-hallucination instructions. The paper's goal is to measure, separately, whether models can extract facts verbatim, reason over them, and stay faithful — and to quantify what is lost when anti-hallucination prompts are applied.

Key Contributions

1. An extended, model-agnostic NIAH benchmark. The framework manipulates context length (up to each model's maximum), information depth, prompt condition, and the statistical distribution of injected facts, and evaluates three capabilities separately: Literal Extraction, Logical Inference, and Faithfulness.
2. Identification of "Distributional Collapse." The authors name and demonstrate a failure mode in which performance degrades because evidence is dispersed or clustered, independent of its absolute position in the context.
3. Quantification of the "Safety Tax." By running every condition under both a Standard Prompt and an Anti-Hallucination Prompt, the paper measures the accuracy lost to over-conservative refusal when the evidence is present.
4. A comparative evaluation of four production-scale models — Gemini-2.5-flash (1M context), ChatGPT-5-mini (272k), Claude-4.5-haiku (~175k), and Deepseek-v3.2-chat (128k) — with automated grading validated against a stratified 60-response human audit that found 98.3% agreement (59 of 60 exact matches).

Main Findings

• Two models stay stable; two do not. Gemini-2.5-flash and Deepseek-v3.2-chat maintained near-perfect accuracy across their entire context range (up to 1M tokens for Gemini). Claude-4.5-haiku's literal extraction degraded notably at its 175k limit, and ChatGPT-5-mini showed a sharp "performance cliff" beyond 100k tokens.
• Aggregate vs. capacity gap. For ChatGPT-5-mini under the Anti-Hallucination prompt, literal extraction fell from 90.3% (aggregate across the bivariate sweep) to 72.0% (conditional mean at maximum token capacity). Under the Standard prompt the same model scored 96.4% aggregate and 89.0% at capacity.
• Claude-4.5-haiku was the weakest overall. Its aggregate Standard-prompt scores were 78.7% literal extraction and 58.5% logical inference; at capacity these fell to 68.0% and 48.0%. It showed a pronounced "U-shaped" degradation, with logical inference dropping to nearly 50% in the middle context intervals before recovering toward the end.
• A positional cliff at the halfway point. ChatGPT-5-mini dropped sharply to approximately 80% accuracy exactly at the 50% depth mark, while Gemini-2.5-flash and Deepseek-v3.2-chat showed high positional robustness across all depths.
• The Safety Tax. Anti-hallucination prompting reduced fabrications but caused over-conservative refusal of facts that were actually present. For ChatGPT-5-mini under Anti-Hallucination prompting, a distinct failure "red zone" emerged at middle-to-late depths once context exceeded 60% capacity. Deepseek and Claude navigated a more balanced trade-off: Deepseek's Anti-Hallucination condition actually raised logical inference from 93.6%/92.0% (aggregate/capacity) to 94.0%/94.0% and faithfulness from 86.7%/84.0% to 91.2%/86.9%.
• Distributional Collapse is severe and distribution-specific. Testing at 100% of each model's capacity, ChatGPT-5-mini's extraction and inference collapsed to 0% under Anti-Hallucination prompts when facts were centrally clustered, as in the Normal and Lorentzian distributions, while it performed adequately on Uniform and Exponential distributions.
• A statistically significant structural failure. Comparing ChatGPT-5-mini's Standard-prompt extraction on the edge-heavy Arcsine distribution (10/10 correct) with its complete failure on the dense Lorentzian distribution (0/10 correct) yielded a Fisher's Exact Test result of p = 1.08 × 10⁻⁵.
• Collapse is not the same as "lost-in-the-middle." Claude-4.5-haiku experienced severe reasoning collapse (≤20%) under the Arcsine distribution, which concentrates information exclusively at the extreme beginning and end of the context and therefore avoids the middle penalty zone. The authors use this to argue Distributional Collapse is an independent flaw, driven by the cognitive load of tracking dispersed facts rather than by absolute coordinates.
• A "Compounding Penalty." When dispersed facts are densely clustered centrally (Lorentzian), maximum cognitive load overlaps with minimum baseline attention, which the authors offer as the explanation for the 0% extraction and inference failures observed in ChatGPT-5-mini.
• Decoupled failure in Claude-4.5-haiku. Under the distributional sweep, Claude retained partial extraction (typically 30–60% accuracy) while logical inference frequently fell to 0% under Uniform and Normal distributions.
• "Faithfulness Masking." Because faithfulness measures only the absence of hallucinations, a model that refuses everything can score perfectly. The paper argues ChatGPT-5-mini's high faithfulness under Anti-Hallucination prompting mirrors exactly the zones where its extraction and inference failed, masking over-refusal rather than demonstrating grounding.
• Nominal context length is a poor proxy for effective context. The authors conclude that effective context length is often significantly shorter than advertised maximums.

Methodology in Plain English

The researchers turned a single-fact haystack test into a two-part stress test.

The haystack. Instead of a synthetic or fact-heavy dataset, they used Honoré de Balzac's La Comédie Humaine (approximately 2M tokens) because its continuous fictional universe lets them inject novel synthetic facts about fictional characters, preventing the models from answering from pre-trained knowledge rather than from the provided text (what the paper calls parametric memory leakage). A "piece-wise Recursive Context Contraction" method summarized segments to reach target token counts while preserving narrative flow.

The quiz. Story-congruent facts ("needles") were injected as paragraphs or sentences, and models answered 30 questions per quiz, graded on a strict binary 0/1 rubric with no partial credit. Every quiz was run under two prompt conditions: a Standard Prompt asking for the most logical answer or reasonable inference, and an Anti-Hallucination Prompt adding the constraint to answer "Not mentioned in the text or story" if the information was neither explicitly nor implicitly present, with the instruction "Don't make it up."

Protocol A swept context length from 10% to 100% of each model's maximum capacity in 10% increments while placing a single fact paragraph at depths from 10% to 100%, producing a 10 × 10 grid and 200 quizzes per model across both prompt conditions. Protocol B fixed context length at 100% of capacity and injected ten fact sentences into 5% bins according to nine distributions — Uniform, Normal, Exponential, Exponential Flipped, Bimodal Gaussian Mixture, Arcsine, Lorentzian, Rayleigh, and Rayleigh Flipped — adding 18 quizzes per model.

Grading. All decoding used deterministic settings: temperature 0.0, top_p 1.0, frequency_penalty 0.0, and presence_penalty 0.3. An independent LLM judge (GPT-5-mini) scored responses against an answer key using the same hyperparameters. The framework reportedly generated 218 quizzes per model, 872 quizzes across four models, and 26,160 individual model responses; the judge's reliability was checked with a stratified human audit of 60 responses (20 each for literal extraction, logical inference, and faithfulness).

Why This Matters

Research impact. The paper reframes long-context evaluation away from a single nominal number (context window size) and toward distributional robustness. It separates extraction from inference from faithfulness, argues that positional bias and Distributional Collapse are distinct phenomena, and offers the pipeline as a reusable, open framework for continuous evaluation as context windows expand.

Real-world applications (as identified in the paper's Impact Statement):

• Legal discovery, where evidence ignored rather than refuted could produce erroneous, authoritative-sounding conclusions.
• Medical record analysis, where dispersed facts across long records may be silently missed.
• Financial auditing, where documents with clustered or sorted evidence could evade LLM-based review.
• High-volume enterprise automation pipelines, which the authors targeted because cost-efficient models are the de facto engines for such workloads.

Adversarial and security relevance. The authors warn that Distributional Collapse could be exploited deliberately: critical information could be hidden from automated audits simply by dispersing it across a document, and the Safety Tax could be weaponized to force over-conservative refusal of valid queries. Their recommended mitigations are to empirically validate candidate models against distributional stress tests before deployment, favoring models that show spatial invariance, and/or to implement hierarchical chunking to break up adversarial fact distributions.

Industry relevance. The paper cautions against replacing retrieval pipelines (RAG) with long-context prompting on the assumption that a bigger window guarantees recall, and argues that selecting robust but smaller production models reduces computational overhead and energy consumption. It also notes an alignment tension: broad, over-conservative refusal mechanisms suppress valid reasoning along with hallucinations.

Future Directions

• Test the architecture hypothesis directly. The authors offer, explicitly as speculation rather than a claim, the possibility that dense attention regimes dilute attention mass while sparse or conditionally-routed computation such as the Mixture-of-Experts design reported for the DeepSeek-V3 family could isolate retrieval and reduce dilution. They state this would require controlled experiments on models whose architectures are fully known.
• Map a prompt severity gradient. The study used only a strict binary contrast between Standard and rigid Anti-Hallucination prompts. The authors hypothesize that softer safety constraints would move behavior along a trade-off frontier, recovering extraction at the cost of increased standard hallucinations, and call the mapping of that optimal gradient a critical open area.
• Extend beyond a uniform narrative corpus. Because La Comédie Humaine lacks nested lists, tables, and domain-specific jargon, the authors characterize their results as likely a best-case scenario and hypothesize that Distributional Collapse is significantly more severe in real-world enterprise documents, triggering catastrophic retrieval failures at much shorter context lengths.
• Evaluate faithfulness jointly with recall. The Faithfulness Masking phenomenon means faithfulness scores must be read alongside extraction recall, otherwise highly "faithful" but functionally paralyzed models may be deployed.

Target Audience

This paper is most useful to practitioners who deploy long-context LLMs in production — engineers building agentic workflows, RAG alternatives, or enterprise document pipelines — and to researchers working on long-context evaluation, hallucination, and safety alignment. It is also relevant to teams evaluating model selection for cost-sensitive, high-volume automation, and to anyone designing benchmarks who wants a template for testing distributional robustness rather than single-fact retrieval. Readers need only an intermediate background: familiarity with LLM prompting and context windows is sufficient, since the framework, prompt templates, grading rubric, hyperparameters, and audit methodology are all documented in the appendices.

Authors’ abstract

As Large Language Models (LLMs) increasingly utilize massive context windows as working memory for autonomous tasks, their reliability fluctuates significantly depending on how information is distributed in real-world corpora. We investigate how fact placement, corpus-level distributions, and anti-hallucination ("Don't Make It Up") prompts influence model behavior by introducing a model-agnostic extended needle-in-a-haystack benchmark designed for scalability, which we apply to evaluate Gemini-2.5-flash, ChatGPT-5-mini, Claude-4.5-haiku, and Deepseek-v3.2-chat. Unlike prior work, we separately evaluate literal extraction, logical inference, and hallucination risk. We identify two critical failure modes: Distributional Collapse, where performance degrades significantly when evidence is dispersed; and a Safety Tax, where anti-hallucination prompts cause over-conservative refusal of present facts and evidence, sharply reducing accuracy. Our results suggest that many failures stem from ineffective context utilization, as models struggle to prioritize relevant information even when it is present. These findings highlight the need for model-specific robustness and effective context management to ensure reliable deployment in long-horizon agentic workflows.