Why Pay for Long Reasoning When Rewrite Do Trick? Causal Effects of Reasoning History Compaction on Future Generation and Agent Cost

AI summary

Overview

Research area: Natural language processing, specifically inference-time efficiency for reasoning language models and LLM-based software agents.

Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with reasoning models, tool-calling agents, token accounting, and bootstrapped confidence intervals.

Scope (one sentence): The paper measures whether rewriting a model's own earlier reasoning traces into shorter "state records" — without retraining and without instructing the model to be brief — causally reduces the reasoning tokens it generates later, and what that costs or saves in real software-agent deployments.

What This Paper Is About

Reasoning models produce long chains of intermediate thought that improve problem solving but dominate inference cost; Artificial Analysis data retrieved on August 3, 2026, cited in the paper, attributes 84.9% of generated tokens to reasoning for GLM 5.2, 75.1% for Claude Fable 5, and 53.4% of total API cost for DeepSeek V4 Pro. Proprietary models appear to reason more compactly, but their scratchpads are hidden, so the author instead studies open-weight models whose reasoning can be observed and rewritten. The core question is whether changing the form of a model's own prior reasoning — compressing it into a concise state record that preserves decisions, uncertainty, and intended actions — acts as an in-context cue that makes subsequent reasoning shorter, and whether that survives contact with real agent budgets.

Key Contributions

1. A reasoning-only compaction intervention with outcome-blind timing. A compressor LLM rewrites historical assistant scratchpads before the pending tool result is revealed, while system messages, user messages, tool calls, arguments, and observations are left intact and the target model's weights and instructions are unchanged. Because results are withheld, neither the rewriter nor the acceptance judge can see future outcomes.

2. A paired-fork causal design with explicit acceptance gates. Each accepted source history is forked into original (clean) and rewritten (compact) histories, independently continued under the same model/provider route and settings. First requests are verified field by field to differ only in assistant reasoning, and the reported estimate is explicitly conditional on accepted treatment histories.

3. A controlled evaluation across six model/provider routes and 12 multi-stage deterministic tasks, yielding 58 retained source histories out of 72 attempted, 174 quality pairs, and 172 comparable token pairs, with source-clustered bootstrap intervals.

4. A deployment study on a fixed 50-task SWE-bench Verified Mini sample with four targets and either an external compressor (GPT 5.6 Luna) or self-compaction, reporting spend, tool calls, target request time, and rewrite-inclusive time, plus an exploratory compressor-effort probe.

Main Findings

• Shortening is frequent but not universal. Compaction shortens 151 of 172 comparable comparisons (87.8%). Route medians range from −20.2% (GLM 5.1) to −50.5% (GLM 4.7 Flash). Twenty pairs grow longer and one is unchanged; the median absolute change is −178 tokens, and the largest increase is 2,807 tokens.

• Route-level effects vary in size. Median reductions with 95% source-clustered intervals: DeepSeek V4 Flash −29.5% [−39.5, −16.5]; Laguna S 2.1 −46.2% [−56.3, −29.6]; MiniMax M3 −27.3% [−38.4, −17.1]; GLM 4.7 Flash −50.5% [−66.6, −40.1]; GLM 5.1 −20.2% [−28.6, −12.5]; DeepSeek V4 Pro −32.6% [−43.4, −12.2].

• Aggregate task success is essentially flat, but not uniformly. Exact success moves from 165/174 to 166/174. GLM 4.7 Flash gains three successes, DeepSeek V4 Flash and Laguna S 2.1 each lose one, and other routes are unchanged. The author notes that only the most heavily filtered route gains, and that this does not establish quality preservation.

• Retention is asymmetric across routes. DeepSeek V4 Flash and Laguna S 2.1 retain 12/12 source histories, MiniMax M3 and GLM 5.1 retain 10/12, DeepSeek V4 Pro retains 9/12, and GLM 4.7 Flash retains 5/12 — which the author says precludes a matched-task model ranking.

• Later turns show larger reductions, but are confounded. In the initial DeepSeek/Laguna comparison of 72 pairs, pooled token reductions grow from 8.8% on turn one to 56.1% on turn three; later turns also reflect treatment-induced changes in previously generated text and actions.

• Encouraging the model to be brief is not the mechanism. The target receives no new instruction to be brief; only the form of its earlier reasoning changes. A length-matched paraphrase control (90–110% of original word count) across 65 comparable pairs yields a median change of +0.7% (95% CI [−8.2, +9.7]) while exact success falls from 68/69 to 59/69, giving little evidence of systematic shortening from generic rewording — though the quality loss means history length is not isolated.

• More compact styles associate with shorter future reasoning, loosely. Across five history styles on ten DeepSeek sources with three continuation blocks each, median changes versus clean are −24.5% (relaxed prose), −27.0% (compact full sentences), −30.1% (telegraphic states), and −37.4% (ultra-telegraphic states). Only 6/30 blocks are monotonic across all tiers, and two of three adjacent contrast intervals include zero; the median Spearman correlation between history word count and future reasoning tokens is 0.70.

• Verification behavior changes in a coding screen. On a LiveCodeBench v6 screen with DeepSeek V4 Flash and Laguna S 2.1, 29/33 comparable pairs shorten, median −57.1%, with both arms solving 33/36 continuations. Extended verification (retrospectively defined as at least 350 reasoning tokens on the submit turn) occurs in 11/18 successful DeepSeek clean branches (61.1%) and none of the compact branches; median final-turn reasoning falls from 489 to 12 tokens.

• Agent deployment economics are mixed. Six of eight compaction configurations cost less and seven reduce target request time (Table 3), but only the two MiniMax configurations reduce rewrite-inclusive request time. Kimi K2.6 self-compaction shows a small inclusive time increase rounding to zero, and Nemotron 3 Ultra is the only target whose spend rises (+9.4% external, +7.2% self).

• Shorter reasoning can coexist with more steps. Kimi K2.6 with external compaction raises selected assistant responses from 1,842 to 2,304 (+25.1%) while mean reasoning per response falls 31.1% and total reasoning falls 13.8%. Observed success differs from Normal by at most three of 50 tasks across configurations, and tool calls change between −29.8% and +19.8%.

• The shortening result survives robustness checks, the quality sign does not. First-attempt (pre-retry) captures give 149 of 169 completed equal-horizon pairs shortening with a paired median of −32.1%. Leave-one-route-out medians stay between −36.7% and −29.2%, but omitting GLM 4.7 Flash reverses the aggregate quality change from a net gain to a net loss.

Methodology in Plain English

The setup has three moving parts. First, a set of deterministic multi-stage tasks (for example, a rescue-planning task with cost, site-count, reliability, and risk constraints, and a superseding weather update) forces a model to make successive tool calls under hard constraints. Second, a compressor model (GPT 5.6 Sol) rewrites the target's historical reasoning into a compact state record, instructed to preserve decision-relevant facts, numbers, uncertainty, conclusions, rejected alternatives, evidence status, and intended next actions, while preserving raw notation rather than translating it. Third, the rewrite is produced after the model requests an action but before that action's result is executed or revealed, so the compressor is outcome-blind.

Accepted source histories are then forked: the original history and the rewritten history are each continued independently under the same model, provider route, and settings. Serialized first requests are checked field by field to confirm only assistant reasoning differs. Because the first turn is identical apart from reasoning, that contrast is a clean measure of the intervention; later turns can also reflect the model taking different actions as a consequence.

Acceptance has mechanical gates (hashes, reasoning-only differences, preserved actions, candidate identifiers, hard thresholds) plus semantic review by a separate GPT 5.6 Sol call using a fixed JSON rubric, with future results excluded from the review payload. Model-based acceptance is explicitly not treated as independent semantic equivalence, and a small human audit by two nonexpert raters (family members of the author) covers six easy pairs and six longer, post-start-selected pairs, reporting overall-label agreement of 4/6 and 5/6 respectively.

Statistics use 10,000 source-level bootstrap resamples, keeping all replicates from a sampled model–task source together, with intervals at the 2.5th and 97.5th percentiles. The headline estimate is conditional on accepted treatment histories and uses equal post-fork turn counts for token comparisons.

The deployment study is a retrospective reanalysis of a fixed 50-task SWE-bench Verified Mini sample. Before each target request, newly generated reasoning blocks are compacted and cached using only their text and action name; blocks of at most 12 words stay unchanged and non-shorter rewrites revert to the original. Accounting links saved target responses and their rewrites to the same evaluated run, including unsolved runs and model-limit exits, and excludes earlier abandoned attempts. Spend includes rewriting; the main table reports target request time excluding rewriting, tools, orchestration, and evaluation.

Why This Matters

Impact on research. The paper reframes reasoning efficiency as a property of history form rather than something requiring retraining or a new decoding-time brevity instruction. It also establishes a clean causal template — rewrite before the pending result is revealed, fork, verify first requests differ only in reasoning — that other groups can reuse. The negative and mixed results are as useful as the positive ones: shortening is robust across leave-one-route-out checks, while aggregate quality change flips sign when GLM 4.7 Flash is excluded, and two of three adjacent style-contrast intervals include zero.

Real-world applications:

• Coding agents and IDE assistants running long tool-use sessions, where prior scratchpads are repeatedly charged as input history — the SWE-bench Verified Mini results show one target's history words dropping 78.9% under self-compaction.
• Cost governance for multi-agent pipelines, where a cheaper external compressor (GPT 5.6 Luna) can be swapped in; six of eight tested configurations reduced spend.
• Long-horizon agent memory and context management, since compaction targets only reasoning while preserving system messages, tool calls, arguments, and observations verbatim.
• Latency-sensitive deployments, with the caveat that target request time and rewrite-inclusive time diverge: the paper notes the hosted one-use preprocessing example loses both time and money.

Industry relevance. The paper makes an explicit business case that compaction is not automatically a win. Rewriting can cost more than it saves — a self-compressor may generate substantial reasoning before returning a short block — and additional steps afterward (Kimi K2.6 external rose from 1,842 to 2,304 assistant responses) can consume the savings. The recommendation is a per-model policy evaluated on task success, total cost, and time spent on both rewriting and task completion, rather than inference from shorter traces alone.

Future Directions

1. Cheaper and less frequent rewriting. Test rewriting every N agent turns rather than before every request, and give the compressor more of the conversation instead of only the block and its action name, trading context for information retention.

2. Cache reuse when the target compresses itself. The paper suggests shared conversation prefixes between target and compressor may allow reuse of cached computation, which should be compared at similar task quality while counting rewrite cost and time.

3. A matched compressor-effort study. The exploratory probe on twenty selected slow DeepSeek requests (historical xhigh 90.1 s median, 12;13 Sol Pass/20; none 1.7 s, 11; low 9.1 s, 10) is confounded by provider and possible checkpoint differences, motivating a larger comparison on the same checkpoint and provider, sampled beyond unusually slow requests.

4. Establishing where the shortening comes from — and missing baselines. The author states the mechanism is unresolved (rewrites may remove repetition, shift emphasis, or drop important doubts), and that the work lacks a matched deletion, truncation, or extractive-retention baseline, so value over free alternatives remains untested. Quality noninferiority is explicitly unestablished, and training-data diversity's effect on model-family compatibility is untested.

Target Audience

Researchers and engineers working on reasoning-model efficiency, inference-time interventions, and LLM agent infrastructure. It is most useful to practitioners who deploy tool-using agents at scale and need to reason about total cost rather than per-request token counts, and to methodologists interested in outcome-blind, paired-fork designs for causal claims about context interventions. The limitations section is written for readers who will scrutinize the deployment accounting and the small nonexpert human audit rather than take headline reductions at face value.

Authors’ abstract

Reasoning substantially improves language models' ability to solve difficult problems, but long reasoning traces make these gains expensive. Efficient reasoning should express useful analysis in fewer tokens by reducing repetition and text that adds little value to solving the problem. Progress in proprietary models motivates this goal, yet their hidden scratchpads limit direct study of how it is achieved. We investigate whether open-weight models can generate less verbose reasoning by changing the form of their own earlier thoughts, without retraining. Our intervention rewrites prior reasoning into compact state records before pending tool results are revealed, then compares continuations from original and compact histories. Across the tested models, accepted compact histories usually shorten future reasoning; a coding case study also shows less repeated verification after a shared candidate solution. In software-agent deployments, compaction can reduce total spending, although task quality and rewrite-inclusive time depend on the target and compressor. These findings identify reasoning history as one way to reduce future reasoning-token use and motivate model-specific compaction policies evaluated for both efficiency and retained capability.