ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models

AI summary

Overview

Research area: Efficient inference for large language models, specifically speculative decoding for hybrid linear attention (HLA) architectures.

Technical level: Advanced. The paper assumes familiarity with speculative decoding, KV caches, linear attention recurrences (Gated DeltaNet), and low-rank projections.

Scope (one sentence): The paper introduces ReaDy, a speculative decoding drafter for hybrid linear attention models that retrieves position-specific context from the target model's fixed-size recurrent states, reporting a 3.16× average speedup over autoregressive decoding with Qwen3.8-27B on a single NVIDIA B200 GPU across seven benchmarks.

What This Paper Is About

Speculative decoding speeds up text generation by having a small drafter model propose several tokens at once for the large target model to verify, but existing feature-based drafters reuse target hidden features that were shaped for the observed position and may not carry the right context for later positions in a draft block. Hybrid linear attention models keep compact recurrent states that summarize the whole prefix in fixed-size matrices, which the authors argue is a cheaper and complementary source of context than scanning a token-wise KV cache. The goal is to let every draft position query those recurrent states directly, improving how many proposed tokens survive verification without adding much computation or memory traffic.

Key Contributions

1. A new context source for drafting. The authors claim to be the first to use HLA recurrent states as a complementary context source for speculative drafting, and provide a theoretical analysis of when a new query to a fixed state reveals information that existing readouts do not determine.
2. The ReaDy method. Built on DSpark's parallel backbone and sequential prediction head, ReaDy adds a lightweight state reader that forms a learned query from each draft representation, reads selected committed recurrent states, and injects the retrieved context through a residual update in the final draft layer.
3. Efficiency engineering. Low-rank state reads (rank 8 over 128-dimensional key and value spaces), fused gather and preparation operators, and a two-stage adaptation strategy keep the extra computation and I/O manageable; operator latency drops from 16.388 to 4.622 microseconds (71.8%) for state gathering and from 65.519 to 47.094 microseconds (28.1%) for state reading.
4. An empirical study of the design space. Ablations cover reader rank, which target layers supply states, how many draft layers receive injections, and where in the layer the residual is applied, plus a fine-tuning control that separates gains from adaptation versus gains from state access.

Main Findings

• Throughput. Mean decoding throughput reaches 302.78 tokens/s for ReaDy versus 280.42 for DSpark and 95.86 for autoregressive decoding, a 3.16× speedup over autoregressive decoding and a 7.97% gain over DSpark.
• Acceptance length. Mean acceptance length rises to 4.5285 from DSpark's 4.1739, an 8.50% improvement.
• Consistency across tasks. ReaDy beats DSpark on all seven benchmarks: +8.66% on GSM8K, +6.43% on MATH-500, +7.76% on HumanEval, +7.28% on MBPP, +6.86% on LiveCodeBench, +9.45% on MT-Bench, and +9.83% on Alpaca.
• Per-benchmark speedups over autoregressive decoding. 4.08× on GSM8K, 3.68× on MATH-500, 3.25× on HumanEval, 3.30× on MBPP, 2.48× on LiveCodeBench, 2.67× on MT-Bench, and 2.63× on Alpaca.
• Better survival deeper into the block. At the seventh draft position, cumulative prefix survival improves from 45.98% to 58.17% on GSM8K, from 25.89% to 31.19% on HumanEval, and from 12.59% to 16.11% on MT-Bench.
• Adaptation alone is not the whole story. A fine-tuned DSpark without the state adapter reaches 286.50 tokens/s and acceptance length 4.2732; ReaDy adds a further 5.68% throughput and 5.97% acceptance length over that no-state variant.
• Rank 8 is the practical choice. Raising reader rank from 8 to 128 lifts acceptance length from 4.5285 to 4.5404 but lowers the throughput gain from 7.97% to 7.81%.
• Layer choice matters a lot. Layer 62 alone gives a 7.48% acceptance gain, versus 2.45% for layer 2 and 2.85% for layer 30; combining layers 30 and 62 gives 8.50%, while adding layer 2 drops it to 6.89% and throughput gain to 6.22%.
• One injection is best. Injecting in only the final draft layer yields 302.78 tokens/s, versus 299.93 for layers 0, 2, and 4 and 297.35 for all five layers, with acceptance lengths of 4.5285, 4.5288, and 4.5284 respectively.
• Post-attention placement wins narrowly. Injecting after attention gives acceptance length 4.4993 to 4.5285 and throughput 301.94 to 302.78 tokens/s compared with pre-attention injection.

Methodology in Plain English

The team starts from an existing speculative decoding drafter, DSpark, which proposes seven-token blocks using a parallel backbone plus a small sequential head, and initializes from a publicly released pretrained DSpark checkpoint for Qwen3.8-27B. They keep that architecture and add a small "reader" in the last draft layer. For each position in the block, the reader takes the draft's internal representation, compresses it through a narrow rank-8 bottleneck, and uses it as a query against recurrent states borrowed from two target layers (30 and 62). The states themselves are compressed along their value dimension to rank eight, so the readout is cheap, and the retrieved vector is concatenated with log-scale statistics about the state and passed through a small MLP whose output is added back to the draft representation as a residual. Because only states belonging to already-committed tokens are read, rejected continuations never contaminate later rounds. Training happens in two stages on the Open-PerfectBlend mixture: 20k samples training only the reader with the drafter frozen, then 50k samples jointly training the reader and LoRA parameters on the drafter, using a cross-entropy plus 0.1-weighted L1 loss and position weights of exp(−j/4). They also rewrote the memory-access path with fused kernels so gathering and preparing states does not materialize intermediate tensors. Evaluation uses SGLang on one NVIDIA B200 GPU in BF16, greedy decoding at temperature zero with thinking enabled, a 1,024-token output limit, and a 16,384-token context limit, comparing against both autoregressive decoding and the original DSpark checkpoint.

Why This Matters

Speculative decoding gains are often eaten by the cost of giving the drafter enough context; this paper shows that for hybrid linear attention models there is a compact, fixed-size memory that can be queried instead of a growing KV cache. That reframes an architectural side effect of linear attention as a usable inference-time resource, and the ablation showing that layer 62 is far more useful than layers 2 or 30 suggests which parts of a hybrid stack carry retrievable information.

Real-world applications:

• Low-latency chat and assistant serving, where the reported MT-Bench gain of 9.45% in throughput translates directly into lower per-request latency.
• Code completion and generation tools, where HumanEval, MBPP, and LiveCodeBench all show throughput gains between 6.86% and 7.76%.
• Mathematical and reasoning assistants, where GSM8K and MATH-500 show the largest absolute throughputs (393.60 and 353.23 tokens/s).
• Cost reduction in inference fleets, since the method runs on a single NVIDIA B200 GPU and requires no quantization and no change to the target model's architecture.

Industry relevance is direct: the work targets production serving stacks (it is implemented in SGLang), builds on an openly released drafter checkpoint, and its engineering contributions are kernel-level optimizations for memory movement rather than only modeling changes.

Future Directions

• Training the complete drafter from scratch rather than adapting a pretrained DSpark checkpoint, which the authors list as future work.
• Learning to allocate state access and readout capacity across draft positions instead of using a single uniform reader configuration.
• Testing whether the reported layer-62 advantage and the two-layer (30, 62) choice generalize to other hybrid linear attention targets, given that the study uses only Qwen3.8-27B.
• Extending the evaluation beyond greedy decoding at temperature zero and a fixed draft block size of seven, both of which are held constant here.

Target Audience

Researchers and engineers working on LLM inference acceleration, speculative decoding, and linear or hybrid attention architectures. It is most useful to readers who already understand KV caching and draft-and-verify decoding and want to know how to exploit recurrent state memory in a serving system; the theoretical appendix and the ablation tables make it valuable for teams deciding where to place a state reader and how much rank to spend on it.

Authors’ abstract

Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven benchmarks spanning mathematics, code generation, and dialogue, ReaDy achieves an average $3.16\times$ decoding speedup over autoregressive decoding with Qwen3.8-27B on a single NVIDIA B200 GPU. It outperforms DSpark on every benchmark, improving throughput by 7.97% and acceptance length by 8.50% on average. Our code is available at https://github.com/StartLuxLabs/ReaDy.