Adversarial Speculative Reasoning Improves Long Horizon Agents

AI summary

Overview

• Research area: Efficient inference for large language models (speculative decoding) applied to long-horizon autonomous agents; intersection of serving-systems numerics, preference optimization, and agentic software engineering.
• Technical level: Advanced. The paper assumes familiarity with speculative decoding, rejection sampling, KV caches, quantization, and direct preference optimization (DPO).
• Scope (one sentence): The paper measures how numerical precision varies across positions inside a speculative verification block, then trains draft models—via reverse-direction DPO—to be rejected at an agent's critical decision points so that those decisions are written by the target model at its least noisy slot.

What This Paper Is About

Speculative decoding is normally treated as a pure speed trick: a small draft model proposes tokens, the large target model verifies them, and a rejection-sampling rule guarantees the final output still follows the target's distribution. The authors argue this "lossless" view hides something important for long-horizon agents, because the target's logits are not bitwise-reproducible: noise grows with each position (slot) inside a verification block, and where a token lands in that block determines how precisely it is written into the model's context memory. Their goal is to exploit this—by training the draft model to deliberately "get it wrong" at the few tokens where an agent commits to a decision, forcing the target model to sample those tokens itself under its most precise conditions—without changing the target model, the verification rule, or the output distribution.

Key Contributions

1. Precision is not uniform inside a verification block. The authors show, on two frontier open-weight models with different draft architectures, that speculative verification is exact in distribution but not in numerical precision: noise grows roughly 4% per slot relative to slot 0, and the first slot is exactly as precise as ordinary single-token decoding.

2. The draft model is reframed from an accelerator into an "allocator of context precision." Because draft acceptance decides which tokens are committed via low-noise residual sampling versus noisier in-block verification, the draft has a control channel that standard training (which optimizes only acceptance rate) ignores.

3. Reverse-direction preference training at decision points. The authors build "fork pairs" of alternative decision framings and train the draft with DPO to prefer the framing the target should avoid, so that at inference the draft and target diverge exactly there and verification restarts at the precise head of a new block.

4. End-to-end gains on agentic coding with no change to the target model. On DeepSWE, GLM-5.3-nvfp4 rises from 60.6% to 67.6% (+7.0 points) and DeepSeek-V4.1-Flash from 67.8% to 71.8% (+4.0 points), while training only 0.06% to 0.28% of the hosted (target-plus-draft) parameters.

Main Findings

• Slot 0 is as precise as decoding; later slots degrade. On GLM-5.3 with an 8-token DFlash verification block, slot-0 noise was 0.473 nats versus 0.478 nats for single-token decoding, then rose at +0.0195 nats per slot (95% CI [+0.0174, +0.0214]), about 4% of the slot-0 level per slot, leaving the last slot 34% noisier than the first.

• Large deviations grow faster than the mean. On GLM-5.3, shifts beyond 2 nats rise from 3.2% of tokens at slot 0 to 6.2% at slot 7, and shifts beyond 3 nats from 1.1% to 2.1%.

• The pattern replicates on a different draft architecture. DeepSeek-V4.1-Flash with a 6-token DSPARK block shows about half the absolute noise of GLM-5.3 but the same relative growth of +0.0094 nats per slot (95% CI [+0.0078, +0.0110]); shifts beyond 1 nat rise from 3.1% to 4.5%, and beyond 2 nats from 0.41% to 0.68%.

• The noise can surface far downstream. Writing a long reasoning turn through verification versus decoding left the two paths indistinguishable for about 30 turns, after which verify-written context produced a 10 to 20% excess of extreme deviations beyond 4 nats (ratios of 0.98 in turns 11–30, 1.09 in 31–50, 1.16 in 51–70 with CI [0.98, 1.36]; an independent build gave 1.20 with CI [1.03, 1.40] beyond turn 30).

• Small log-probability shifts change sampling outcomes. A 1–2 nat shift moves the emitted token in or out of the sampling nucleus in 12% of cases, 2–3 nats in 47%, and more than 3 nats in over 84%.

• How a decision is framed predicts whether it is right. On GLM-5.3, decisions made in a "principle" or "evidence" frame were correct about 70% of the time, versus 36% for "likely" and "risk" frames.

• Drafts can learn framing and behavior, but not correctness. Held-out preference accuracy reached 0.635 (±0.015 seed spread) for adversarial decision frames on GLM-5.3 and 0.85 for the adversarial run-tests preference on DeepSeek-V4.1-Flash. By contrast, "right vs. wrong" stayed near chance (best of 36 forward and 12 adversarial configurations: 0.61 and 0.57 on 92 and 128 pairs; medians 0.52 and 0.46), and thinking-length preferences landed in between (0.54 forward, 0.65 adversarial).

• End-to-end agentic gains. Five independent runs per configuration on DeepSWE (113 tasks, hidden tests) and NL2Repo-Bench (104 repositories graded by upstream pytest): GLM-5.3-nvfp4 gained 7.0 points on DeepSWE (95% CI 2.6 to 11.4); DeepSeek-V4.1-Flash gained 4.0 points on DeepSWE (95% CI −1.3 to 7.3) and 1.4 points on NL2Repo-Bench, 66.4% against 65.0% (95% CI 0.4 to 2.4, p = 0.011).

• Very small trainable footprint. The trainable set is 2.15B parameters for GLM-5.3 (87% of its 2.46B draft, 0.28% of the deployed system) and 449M for DeepSeek-V4.1-Flash (3.2% of its 14.2B draft, 0.06% of the system); both targets are about 750B parameters (753B and 749B respectively). The speed comparison promised in §5.3 is referenced but its numbers are not reported in the provided content.

Methodology in Plain English

1. Measure the noise first. The authors hold the input prefix fixed in GPU memory and repeat only the forward pass four times, so that differences between repetitions come purely from numerical nondeterminism—not from re-reading the prompt. They do this at every slot of a verification block, over 1631 positions on GLM-5.3 and 1294 on DeepSeek-V4.1-Flash, drawn from DeepSWE prompts with 6k–24k tokens of context. A second experiment writes a whole reasoning turn into memory either through block verification or through ordinary decoding, then scores the rest of the real trajectory (up to 128k further tokens, a median of 95 later assistant turns, over 96 tasks and 6.38M scored assistant tokens per build per path).

2. Find the decision points. A judge—the target model itself, not a stronger external model—labels reasoning and returns the "committing sentence" (at most 35 words) where the agent settles on a reading, a plan, or a sign-off. Two preferences are used: decision frame (principle/evidence are preferred; likely/risk/simpler are adversarial) on GLM-5.3, and run-a-test vs. don't on DeepSeek-V4.1-Flash, labeled by the tool call that follows the thinking.

3. Build clean training pairs. For each decision point they keep the shared prefix and resample 16 (GLM-5.3) or 12 (DeepSeek-V4.1-Flash) continuations at temperature 1 with top-p 0.95, then pair a preferred continuation with an adversarial one from the same fork, so that the shared prefix cancels out and the gradient comes only from what happens after the fork. GLM-5.3 yields 1,668 forks (28,356 samples) and 2,843 pairs from 28 tasks with six tasks (619 pairs) held out; DeepSeek-V4.1-Flash yields 2,203 forks (26,418 samples) and 6,391 pairs from 92 tasks, with 10% of tasks (719 pairs) held out and 5,100 pairs used for training.

4. Train the draft backwards. Instead of teaching the draft to imitate the target (the usual acceptance-maximizing objective), they apply DPO so the draft prefers the undesirable side. The reward is a weighted mean log-ratio between the trained and frozen draft, restricted to a window around the decision: 32 tokens before the committing sentence through its end at full weight, plus a 128-token decaying tail (~200 tokens total), with β = 10; the run-tests preference uses the first 512 tokens at full weight.

5. Handle the engineering. Target hidden states are extracted once and the draft is trained offline on those features. The DeepSeek DSPARK draft (about 14.3B parameters, 13.6B of them in FP4 routed experts) had inference-only kernels, so the authors ported it to differentiable PyTorch, matching reference kernels to within 10⁻⁷ per expert and 3–5% per stage output, and trained FP8 weights with quantization-aware training in 32×32 blocks with power-of-two scales.

6. Evaluate with everything else frozen. Two servers differing only in the draft (pretrained vs. adversarial), identical target weights, serving stack, exact verification, sampling parameters, and agent harness; five independent runs each; internet disabled in the development workspace to prevent reward hacking.

Why This Matters

Impact on research. The paper challenges the assumption that speculative decoding is behaviorally neutral, and it introduces a new degree of freedom—where a token is committed—that prior work optimized away. It also connects serving-system numerics (quantized KV caches, sparse attention, non-batch-invariant kernels) directly to agent reasoning accuracy, and shows a way to change model behavior by training well under one percent of parameters while never emitting a token outside the target's distribution.

Real-world applications:

• Coding agents that work for hundreds of turns in a repository, where a single early misreading of a requirement compounds (DeepSWE-style tasks with contexts past 200k tokens).
• Repository-scale generation from specifications, such as building multi-module Python libraries from a requirements document (NL2Repo-Bench) where partial test-pass rate matters.
• Cost-efficient serving of frontier models, since the intervention lives entirely in the draft model and preserves the exactness guarantee of verification.
• Debugging and auditing agent reliability, by giving engineers a way to see which reasoning decisions were committed under the noisiest conditions.

Industry relevance. Any team already running speculative decoding in production can apply this without touching target weights or the verification rule; the method piggybacks on existing draft infrastructure. The finding that low-precision serving choices (NVFP4 weights, FP8 KV caches, sparse attention) interact with agent accuracy also matters for anyone making deployment decisions about quantization.

Future Directions

• Directly test the durability hypothesis. The authors hypothesize that decisions committed at higher precision "weigh more" over a long horizon, but they do not measure downstream retention of individual decisions directly; the paper itself notes that wrong decisions can be flipped by environmental feedback while correct ones can be ignored.
• Extend beyond coding benchmarks. The current evidence covers DeepSWE and NL2Repo-Bench with two models; whether the effect transfers to other long-horizon domains (research agents, computer use, multi-agent systems) is open.
• Probe what drafts can and cannot learn. The reference preferences show a small draft cannot learn semantic correctness ("right vs. wrong" stayed within chance), raising the question of whether larger or differently structured drafts could, and whether other preference types (thinking length reached 0.65 adversarially) are usable.
• Separate the two levers. The paper does not disentangle how much of the gain comes from precision reallocation versus from verification restarts after rejection, and the speed results promised in §5.3 are not reported in the available content.

Target Audience

Researchers and engineers working on LLM inference systems, speculative decoding, and agent infrastructure will get the most from this paper, particularly those who serve frontier open-weight models with quantized weights and KV caches and who run long-horizon agents in production. It is also relevant to agent-reliability researchers interested in how numerical precision and error accumulation shape multi-turn reasoning, and to practitioners of preference optimization looking for an unusual application: DPO applied to a draft model in the reverse direction. Readers without background in speculative decoding or DPO will find the systems sections dense, though the framing in the introduction and Section 4 is accessible.

Authors’ abstract

Speculative decoding has been treated purely as a systems optimization that speeds up large language model decoding without changing the model behavior. Under this view, the growing effort to train better draft models has been focused solely on acceptance rate. We argue that this view does not hold true for long-horizon agent reasoning. Measuring frontier open-weight models on complex agentic tasks with different speculative decoding approaches, we found that numerical noise grows within verification windows, and the draft acceptance results of the thinking process can change the output distribution of the model after 30 steps, thus noticeably influencing the reasoning accuracy of long-horizon agents in random directions. In this work, we improve the reasoning capability of target models by controlling such noise. While we cannot change the overall token distribution verified by the target model, we can encode critical reasoning with highest precision by training the draft model to be rejected. We identify these points in agent traces and train the draft model to favor adversarial preference. At inference the trained draft and the target part ways precisely at these decision points, verification restarts, so that critical thinking tokens are sampled from the target model and encoded under its least noisy conditions. This approach makes critical reasoning better indexed and more durable over a long horizon. On complex agentic coding benchmarks, our approach raises end-to-end success rate with no change to the target model or its output distribution. Our results recast the speculative decoding mechanism from a pure accelerator into an active allocator for context precision that improves the accuracy of long-horizon agents by training 0.06% to 0.28% of hosted parameters.