You don't need long context

AI summary

Overview

Research area: Efficient language-model inference and agent context management (LLM systems, agentic workflows, long-context modeling).

Technical level: Intermediate. The paper combines a serving-cost measurement study with agent benchmark experiments; readers need some familiarity with context windows, KV caches, and agent harnesses, but the arguments are conceptual rather than mathematically dense.

Scope: The paper tests whether "long-context" workloads — both long-horizon agentic tasks and long-input document tasks — can be completed with deliberately short contexts, and argues that long-context models are unnecessary.

What This Paper Is About

Every token a language model generates gets more expensive as the context grows, yet frontier models ship with context windows of 256K tokens and beyond. The authors ask a root question — "Are long contexts necessary?" — and then test it empirically by forcing frontier models to work inside short contexts through repeated compaction (summarizing and restarting the conversation) and through reading large inputs selectively from a filesystem instead of ingesting them whole. The goal is to show that task accuracy survives, and often improves, when context is kept short.

Key Contributions

1. A quantified cost case against long context. The authors measure peak decode throughput for GLM 5.2, Kimi K2.6, and Qwen 3 Coder Next under vLLM on NVIDIA H100 GPUs and convert it to dollars at $3.85 per GPU-hour. Serving at a 256K context window rather than 16K costs 3.6x more per generated token for GLM 5.2, 5.6x more for Kimi K2.6, and 7x more for Qwen 3 Coder Next.

2. Evidence that long-horizon agent tasks survive frequent compaction. Across six long-horizon benchmarks (SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.1, FRAMES, τ²-bench, BFCLv3), End-of-Conversation Compaction (EoCC) keeps accuracy broadly flat while yielding estimated cost savings of 2.2x–8.8x for Claude Opus 4.5 and 2.4x–5.0x for GPT 5.6 Sol.

3. Evidence that long-input tasks do not need the input in context. With documents stored as files and read selectively via Bash tools, seven long-input benchmarks (LongBench v2, LOONG, AA-LCR, NIAH, RULER, Counting-Stars, BABILong) match or beat full-input baselines, with savings of 6.5x–54.6x (Opus 4.5) and 4.4x–233.3x (GPT 5.6 Sol).

4. The Semantic State Space Model (SSSM) proposal. Because Single-Action Compaction (SAC) — exactly one action per context — nearly preserves baseline accuracy, the authors propose models post-trained to carry an explainable natural-language state between turns rather than an ever-growing conversation history.

Main Findings

• Token cost scales with context depth. The paper reports an inherent linear relationship between the computational cost of generating a token and its depth in the context. On the local side, later tokens took approximately 2x longer to generate than earlier tokens for several models measured on an NVIDIA DGX Spark.

• Context windows dwarf the texts they are meant to hold. Table 1 compares model windows to literary works: The Great Gatsby is roughly 64K tokens, War and Peace roughly 753K, the King James Old Testament roughly 802K. Meanwhile MiniMax M2.7 and Claude Opus 4.5 offer 200K, and Gemini 3.8 Flash and GLM 5.2 offer 1M. The authors note that short contexts of 16K–50K already suffice to contain elaborate literary works.

• Compaction is nearly free in accuracy on long-horizon tasks. Across all benchmarks and thresholds, Opus 4.5 ranged from -3.7 to +5.6 percentage points relative to baseline and GPT 5.6 Sol from -3.3 to +4.4. For either model, no more than 5 of 22 threshold configurations lost more than 2 percentage points.

• Comapction did not blow up total token spend. The authors checked whether re-doing work would erase savings; in their experimentation this was generally not the case.

• Long-input accuracy was matched or improved, sometimes dramatically. Opus 4.5 ranged from -0.4 to +40.0 percentage points and GPT 5.6 Sol from +1.0 to +30.9. Conversation length stayed broadly bounded as document length grew, unlike the y = x reference curve.

• One action per context is viable. For Claude Opus 4.5, SAC preserved the mini-swe-agent baseline on SWE-bench Verified (70.0 vs 70.2) and improved Terminal-Bench 2.1 by 4.6 points (47.2 to 51.8). For GPT 5.6 Sol, SAC reached 82.4% on SWE-bench Verified and 84.3% on Terminal-Bench (75 of 89 tasks solved), 2.0 and 3.3 points below its mini baseline.

• SAC shrinks live context drastically. Median peak live context under SAC was only 7–9K tokens, and the maximum fell from 130–200K to 15–37K tokens.

• History can still matter — but should be retrievable, not resident. The authors concede compaction is lossy and construct a coin-toss thought experiment where accuracy must eventually degrade. Their fix is to log every message and expose it through a selective reading tool, turning history access into a long-input task.

Methodology in Plain English

The authors start by measuring what long context actually costs: they serve several large open-weight models on H100 nodes (eight GPUs for Qwen 3 Coder Next, sixteen for GLM 5.2 and Kimi K2.6), prefill a long shared prefix, enable prefix caching, require exactly 1,024 output tokens per request, sweep concurrency, and record sustained decode throughput. They also measure latency and memory for locally run models on a DGX Spark with 128GB of unified memory, about 109GB of which is available for serving.

For accuracy, they take two off-the-shelf agent harnesses — mini-swe-agent for software and terminal tasks, and a generalized mini-agent for everything else — both built on a minimal ReAct loop where the model issues one Bash command at a time. They then compare three ways of managing conversation history. Full-History Accumulation (FHA) is the baseline: everything is appended forever. End-of-Conversation Compaction (EoCC) lets the conversation grow until it crosses a threshold, then asks the model to write a bounded summary and restarts a fresh conversation from that summary, repeating as needed. Single-Action Compaction (SAC) permits exactly one action per conversation, carrying forward only a compact semantic state.

For long-input tasks, they change only how material is presented: documents are written to the agent's filesystem as files such as input01.txt, and the agent uses ordinary tools like sed, awk, and head to read what it needs. This converts a long-input workload into a long-horizon one. They methodically tighten the total context limit and compaction threshold — for example 48K/36K, 32K/24K, 24K/18K, and 16K/12K on SWE-bench Verified, Terminal-Bench 2.1, SWE-bench Multilingual and FRAMES — and plot accuracy change and cost savings against each setting.

Why This Matters

Impact on research. The paper reframes a scaling assumption. Instead of asking how to make models handle longer inputs, it asks whether the inputs need to be in context at all. The SSSM proposal — a model that maintains an explainable natural-language state between turns, contrasted with the black-box neural state of a classical State-Space Model — is a concrete architectural research direction. The authors also argue SSSMs would be more data-efficient to train because the model need not learn which patches of earlier context to attend to. They tie the whole argument to the notion of computability resting on bounded input and finite state.

Real-world applications.

• Cloud LLM serving: operating at 16K rather than 256K context yields 3.6x–7x more decoded tokens per second on identical hardware, a saving the authors say rivals aggressive quantization or structured sparsification.
• On-device and privacy-sensitive deployment: the DGX Spark measurements target cases where source code or task data must stay local; the ~2x inter-token latency penalty and larger memory footprint at long context directly limit what fits.
• Autonomous software engineering: SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench 2.1 show repository patching and command-line work survive compaction and even single-action operation.
• Long-document analysis: financial reports, legal cases, academic papers (LOONG), government consultations and industry reports (AA-LCR) can be handled by selective reading rather than full ingestion.

Industry relevance. Because retail token prices are quoted as a flat price per million tokens regardless of input length, the cost gradient is hidden from end users but very real to providers. The paper argues the benefit accrues directly to whoever serves the model.

Future Directions

• Train an actual SSSM. The paper proposes the concept and defines its five required behaviors but does not report training such a model; building and evaluating one is the obvious next step.
• Test the lossy-compaction boundary systematically. The authors acknowledge their evidence "does not rule out" cases where history is genuinely load-bearing, and offer only the coin-toss argument plus the history-as-retrievable-log workaround.
• Replace naive compaction with better state representations. The compaction used here is described as naive and lossy; learned or structured semantic states that lose less information per compaction are an open design space.
• Validate beyond the tested harnesses and models. Results come from two closed-weight models as principal workers plus open-weight models in Appendices C and D; whether the pattern holds across other post-training recipes and agent scaffolds is not established here.

Target Audience

Machine-learning systems engineers and inference-infrastructure teams who pay for context; agent-framework developers deciding how to manage conversation state; and researchers working on long-context modeling, memory, or state-space architectures who want a well-measured counterargument to the assumption that bigger context windows are progress. Practitioners deploying coding agents or document-QA pipelines on constrained hardware will find the concrete threshold configurations and cost multiples directly actionable.

Authors’ abstract

Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.