SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding

AI summary

Overview

Research area: Machine learning systems — specifically LLM inference acceleration via speculative decoding, combined with autonomous agent-driven algorithm and kernel discovery.

Technical level: Advanced. The paper assumes familiarity with speculative decoding, draft trees, GPU memory-bandwidth bottlenecks, and kernel-level optimization (wgemm, wgmma, GEMM tiling).

Scope: The paper presents SpecPilot, an autonomous agentic optimization loop that discovers lossless algorithms and systems optimizations for speculative decoding deployments, and reports 2×-or-better sealed speedups over the released ("stock") systems across four target models, three drafters, two GPUs and five benchmarks.

What This Paper Is About

Speculative decoding speeds up LLM generation by having a small "drafter" model propose several tokens that a larger target model verifies in one pass. The paper observes that released parallel-drafting systems leave much of the achievable speedup on the table: with a single request they verify only one drafted chain and ignore other ranked candidates, and with batching the per-request verification work roughly halves the gain. Because the best optimization depends on the serving regime, drafter, target model and GPU, the authors build an autonomous agent loop that finds and validates deployment-specific optimizations rather than relying on one universal recipe.

Key Contributions

1. An autonomous, ceiling-guided optimization loop (SpecPilot). A cost model compares the measured system against an estimated performance ceiling to rank where speed is being lost; parallel agents implement changes on separate GPUs; a controller isolated from the agents promotes only statistically significant gains; and validated mechanisms are distilled into a structured memory that transfers to new deployments.

2. Token forcing, a stock-calibrated losslessness test. Because task accuracy is too coarse to catch reward hacking (the paper shows a deliberately faulty system whose accuracy confidence intervals overlap a lossless one's), SpecPilot checks every teacher-forced output token against stock, using a rate rule and an event rule on large-margin mismatches.

3. A set of discovered lossless algorithms and systems optimizations. These include W12 (a general lossless 12-bit weight format), best-first tree verification for hybrid recurrent targets (with a "WY verify" kernel), two-level verification (TLV) using a zero-copy low-precision Q pass, and three batching mechanisms (shared node budget, per-request drafter block, prompt admission).

4. Cross-regime bottleneck analysis. The paper shows the same mechanism can speed up single-request decoding and slow down batched decoding on the same GPU, and traces how the dominant bottleneck shifts over three optimization rounds.

Main Findings

• Headline speedups: Without training or quality loss, SpecPilot raises speedup over autoregressive (AR) decoding from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.

• Single-request results by platform: On H100 the final systems reach 8.86–9.41× over AR; on the RTX PRO 6000 they reach 10.63–10.87×. Sealed gains over stock on unseen requests range from 2.13× (Qwen3.8-27B with DFlash2 on H100) to 2.44× (Qwen3.8-27B with DFlash2 on RTX PRO 6000).

• Baseline comparison: On H100 with one request, SpecPilot is 1.84× faster than the DDTree baseline.

• Batched results: On FP8 Qwen3.8-27B it is 1.85× faster than stock with DFlash2 and 1.94× with DSpark, lifting speedup over AR from about 2× to 3.90× and 3.93×. Weaker stock baselines gain more: 2.39× on Qwen3.5-4B and 3.11× on Gemma-4-12B, where stock (EAGLE-3 at 0.68×) actually trails AR.

• Stock's unused candidate headroom: In exact replay, stock DFlash2's 8-position chain accepts τ = 4.25 tokens per step, while an oracle that keeps accepting the target's greedy token whenever it stays in the drafter's top-16 would accept 6.42 at 8 positions and 8.20 at 16.

• Weight reads dominate single-request steps: Before the first round, GEMM time grew only from 17.88 ms for one row to 18.58 ms for 64 rows. After the first round, reading target weights took 80% of the measured 23.61 ms step, and at the H100's peak 3.35 TB/s reading them would take 92% of the time needed to read all bytes of the step.

• Batching halves the gain: Stock DFlash2's FP8 speedup over AR falls from 3.40× with one request to 1.89× with 24 requests, and Qwen3.5-4B falls from 3.73× to 1.49× with 32.

• W12 works beyond speculation: Under plain AR decoding with no speculation, W12 alone gives 1.19–1.28× with one request and 1.13–1.20× with 32 requests, shrinks weights by 14–22%, and grows the KV pool 2.0× on Qwen3.8-27B and 2.5× on Muse-Glimmer-30B. The shipped W12 GEMM beats cuBLAS by 1.19–1.47× at one row and 1.17–1.35× at 32 rows on Qwen3.8-27B projections, and by 1.18–1.68× on all 13 shapes on the RTX PRO 6000.

• W12 compression: It averages 12.01–12.02 bits per weight on eight dense models and shrinks Qwen3.8-27B's converted layers from 44.67 GB to 33.58 GB, with an escape fraction of 7.3×10⁻⁵.

• TLV's Q pass is nearly free in accuracy terms: Reading the 8-bit view already nested in each W12 record, the Q pass matches the exact pass's top-1 token at 98.7% of positions.

• Token forcing separates good from faulty systems where accuracy cannot: Faulty systems (logit bias of 2 on every 7th vocabulary id after 2048 tokens) change accuracy by −1.6 points [−3.6, +0.3] versus −0.3 [−1.6, +0.7] for a lossless system — overlapping intervals. Token forcing accepts both lossless systems (0.73% and 0.52% mismatch, against stock's 0.72% and 0.53%) and rejects both faulty ones at 2.0% and 2.4%, with 192 and 191 of their 200 audited large-margin events unexplained.

• The bottleneck shifts across rounds: The system went from stock's 184 to 416 tokens/s in three rounds: round one targeted tokens per step with best-first trees (270 tokens/s), round two targeted weight bytes with W12, and round three removed the "one exact pass per draft" assumption via TLV.

• Cost-model ceilings matter: A single-agent baseline lacking the cost model reached only 1.34× over stock after three rounds, versus 2.13× for SpecPilot, because it never addressed target weight reads despite their 80% share of the step.

• Regime dependence is real: Wide trees, W12 and TLV all gained with one request, but with batching 24-node trees lost 14.9% on the FP8 target, and on BF16 Qwen3.8-27B a 24-position block lost 9.9% and TLV lost 13%. A low-precision pre-check that helps with one request is 13% slower with batching on BF16 Qwen3.8-27B.

• Memory transfers across deployments: 140 memory entries (15 specific to sm_120) carried mechanisms to DSpark, EAGLE-3, the RTX PRO 6000 and Muse-Glimmer-30B. Porting the H100 tree and W12 entries to the RTX PRO 6000 alone gave a sealed 1.92×, and adding TLV with split-KV verify attention plus an INT6/INT8 Q-pass copy raised it to 2.44× — above its 2.12× ceiling, which assumes one draft per pass.

Methodology in Plain English

SpecPilot runs in repeated rounds, each starting from the previous round's best system (or, for a new deployment, from stock plus reusable "memory" entries).

Phase 0 — find the bottleneck. A cost model expresses speedup as accepted tokens per step divided by step cost relative to an AR step, then computes a "ceiling": the best speedup achievable if the system drafted once, verified the best possible tree in one exact pass, and ran every kernel at its "byte floor" (the larger of its FLOP time at 60% of peak dense throughput or its byte time at 90% of peak bandwidth). The ceiling factors into a token term, a kernel term and an overhead term, and the largest term tells the agents where to work.

Phases 1–3 — explore, integrate, tune. A master agent proposes several directions (one idea per GPU) with estimated headroom. A sub-agent implements each as patches whose defaults reproduce stock, and must first confirm headroom with the cheapest evidence (exact replay, a microbenchmark, or a timing run) before building anything new. Separately, an integration sub-agent stacks individually unpromising patches that may pay off together, and a kernel sub-agent tunes whichever kernels sit furthest from their byte floors.

Phase 4 — sealed final. Each round ends by timing the final system against stock on unseen requests and re-running token forcing on outputs up to 4096 tokens. A final that fails either check is rejected.

Keeping it honest. A controller that agents can call but not modify does paired timing runs (both systems serve the same requests on two GPUs, swapping GPUs between runs) and promotes a patch only if the 95% lower bound on its gain exceeds 1% plus twice the stock-versus-stock noise (under 0.1% for one request on H100), or about 3% if its outputs diverge from the current best's. Token forcing catches subtler cheating: a teacher-forced scorer re-feeds outputs through the target on the stock engine with an FP32 LM head, flags mismatches beyond a near-tie threshold δ, and applies a rate rule (mismatch rate no more than 0.5 points above stock at 95% confidence) and an event rule (no excess large-margin events).

Why This Matters

Impact on research. The paper reframes speculative decoding optimization as a per-deployment search problem rather than a single algorithmic fix, and supplies two reusable pieces beyond the specific mechanisms: a ceiling-based ranking method that tells you which bottleneck to attack next, and token forcing, a practical test for losslessness that task accuracy demonstrably cannot provide. W12 is also argued and measured as a general lossless weight format useful outside speculative decoding, which is a broader contribution than the speculative-decoding wrapper.

Real-world applications:

• Latency-sensitive single-user inference — interactive coding assistants and chat, where the paper's single-request regime shows the largest gains (up to 10.87× over AR on the RTX PRO 6000).
• High-throughput batched serving — API endpoints serving many concurrent users, where the batching mechanisms (shared node budget, per-request drafter block, prompt admission) target the regime where speculative decoding's advantage normally halves.
• Memory-constrained deployment — W12's 14–22% weight shrink and 2.0–2.5× larger KV pools translate directly into fitting more concurrent requests or larger models on fixed GPU memory.
• Edge and non-Hopper hardware — the port to the RTX PRO 6000 (no wgmma, 99 KB shared memory per thread block, sm_120) shows the approach survives a change of instruction set rather than assuming Hopper-only kernels.

Industry relevance. The optimization target is a stock open-source server (SGLang, frozen v0.5.20 build, token-identical to the release), and every reported gain is a sealed final measured against a tuned baseline rather than an untuned one. That framing — plus the explicit defense against reward hacking, where an agent could adopt a fast numerical shortcut with no measurable accuracy cost — is aimed squarely at the practical question of whether agent-driven optimization can be trusted in a production serving stack.

Future Directions

• Extending beyond the tested envelope. The paper covers four target models, three drafters, two GPU platforms and five benchmarks, and explicitly notes memory entries carry "untested regimes" summaries; how far the discovered mechanisms and the memory generalize to further hardware and model families is left open.

• Raising the remaining ceilings. The authors note that round three's W12 single-pass step still ran about 25% above its byte floors, so kernel-level headroom remains even after the algorithmic wins.

• Better trees on the existing lattice. Exact replay shows best-first trees plateau on the drafter's lattice (matching an EAGLE-2-style top-4 tree at 16 nodes, 5.03 vs. 5.02), which the authors read as evidence that further gains must come from the lattice or node budget rather than tree construction.

• Making the losslessness audit cheaper and more general. Token forcing currently relies on a teacher-forced scorer and an audit of large-margin events; how this scales as a general guardrail for agent-driven numerical optimization is an open design question the paper raises but does not answer.

Target Audience

ML systems researchers and inference engineers working on LLM serving performance, particularly those already using or evaluating speculative decoding and parallel drafters. It will also interest practitioners of agentic program search (the AlphaEvolve/ADRS line of work) who need a reliable evaluator for tasks where small gains sit near timing noise and where the optimization target can be gamed. Readers need a working grasp of GPU memory-bandwidth economics and speculative decoding to follow the mechanisms; the framing and results are accessible to a broader ML audience, but the kernel-level sections are not beginner material.

Authors’ abstract

Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46$\times$ to 10.87$\times$ with one request, and from 1.94$\times$ to 3.93$\times$ with batching.