3-Model Speculative Decoding

AI summary

Overview

Research area: Natural Language Processing / large language model inference efficiency (speculative decoding).

Technical level: Intermediate. The paper assumes familiarity with speculative decoding, logits, and token acceptance rates, but its core ideas are explained conceptually and the math is limited to a few throughput formulas.

Scope: The paper proposes Pyramid Speculative Decoding (PyramidSD), a three-model decoding framework that inserts an intermediate "qualifier" model between a small draft model and a large target model to raise token acceptance and speed up LLM generation, evaluated on the CSQA benchmark using the Llama 3.2/3.1 model family on RTX 4090 GPUs.

What This Paper Is About

Standard speculative decoding (SD) speeds up LLM inference by having a small draft model propose tokens that a larger target model then verifies. The problem is a trade-off: the smaller the draft model, the faster it runs, but the more its predictions diverge from the target's, which lowers the acceptance rate and erodes the speedup. PyramidSD addresses this by placing a mid-sized qualifier model between the draft and target, so tokens are checked twice — first by the qualifier, then by the target — closing the distributional gap and allowing much smaller draft models to be used.

Key Contributions

1. A novel three-model decoding framework that layers two speculative stages (draft to qualifier, qualifier to target) to accelerate inference using an intermediate qualifier model.
2. Two extensions of the speculative decoding acceptance criterion — the fuzzy-acceptance variant (PSD_F) and the assisted-decoding variant (PSD_A) — that allow a fine-grained trade-off between performance, stability, and speed.
3. Experimental demonstration that PyramidSD decodes faster than standard SD while maintaining relative performance, with a reported peak of up to 1.91× generation speed over standard SD and 124 tokens per second on an RTX 4090 consumer GPU.
4. A quantitative acceleration analysis showing how speculative lengths and fuzzy divergence thresholds interact to determine efficiency, including the finding that non-monotonic performance patterns arise from the interplay of thresholds and speculative lengths.

Main Findings

• Speedups over standard SD: PyramidSD with assisted decoding (PSD_A) provides up to 1.44× improvement over SD with low variance across runs; PyramidSD with fuzzy acceptance (PSD_F) delivers up to 1.91× faster than SD but with larger standard deviation.
• Quality on CSQA: Standard SD scores 69.58 ± 2.20; Fuzzy SD (FSD) shows mild sensitivity to the divergence threshold τ_T, averaging 70.03. PSD_A tracks FSD closely across thresholds (e.g., 70.73 ± 2.57 at τ_T = 0.4; 68.55 ± 2.59 at τ_T = 0.5), while PSD_F shows higher variance (e.g., 72.31 ± 8.85 at τ_Q = 0.2, τ_T = 0.2; 63.50 ± 13.20 in one configuration).
• Threshold ordering matters: Performance is best maintained when τ_Q ≤ τ_T, so the qualifier filters clear mismatches first and the target performs final verification efficiently. Overly permissive thresholds (τ_T > 0.5) degrade output quality without proportional speedups.
• Entropy gradient across model scales: The 1B draft model produces relatively uniform entropy distributions (high uncertainty), the 3B qualifier produces sharper distributions with retained flexibility, and the 8B target shows strongly right-skewed, highly confident predictions. The same hierarchy appears in maximum predicted token probability at each decoding step.
• Speculative length balance: Larger qualifier lengths (e.g., ℓ_Q = 25) generally yield higher throughput, but increasing ℓ_D does not uniformly help — beyond a point acceptance drops. Setting ℓ_D to approximately half of ℓ_Q consistently gives the best trade-off, with optimal ratios typically between 1:2 and 1:3.
• Non-monotonic behavior: More aggressive speculation — higher thresholds, longer draft sequences, or deeper relaxation — does not always yield higher throughput; medium-threshold, moderate-length configurations sometimes outperform extreme setups due to rejection cascades.
• Measured throughput range: Vanilla SD in the ablation table ranges from 42.03 to 65.01 tokens/sec across speculative lengths; PSD_A configurations range from around 30.89 up to 93.71 tokens/sec; PSD_F configurations reach as high as 115.86 tokens/sec in the reported tables, with the abstract citing 124 tokens per second on an RTX 4090.
• PSD_F instability: The multiplicative nature of acceptance rates across two stages means errors from the draft-to-qualifier stage can propagate downstream, amplifying output fluctuations; PSD_A avoids this by sampling from the qualifier's distribution when draft tokens are rejected, providing a guaranteed quality floor matching the qualifier.

Methodology in Plain English

The researchers took an existing approach — fuzzy speculative decoding, which relaxes strict token equality by allowing acceptance when output distributions are sufficiently similar under a divergence threshold — and extended it to three models instead of two. They chose a Llama 3.2 1B model as the draft, a Llama 3.2 3B model as the qualifier, and a Llama 3.1 8B model as the target, all instruction-tuned and sharing a tokenizer and vocabulary. The draft proposes ℓ_D tokens, the qualifier verifies them under threshold τ_Q, and the resulting ℓ_Q tokens are passed to the target for final verification under threshold τ_T. They evaluated four strategies — standard SD, fuzzy SD, and two PyramidSD variants — on the CSQA evaluation set, setting maximum generation length to 2048 tokens and sampling temperature to 0.7, with five pre-determined QA samples per prompt, reporting mean and standard deviation. Hyperparameters were chosen by grid search: τ_Q, τ_T ∈ {0.2, 0.3, 0.4, 0.5}, ℓ_D ∈ {2, 4, 6}, and ℓ_Q ∈ {1, 2, 3, 4, 5, 7, 10, 15, 20, 25}. All experiments ran on RTX 4090 GPUs with 24GB of memory, built on the Hugging Face Transformers implementation. They also analyzed model behavior by measuring entropy and peak token confidence across the three model sizes.

Why This Matters

Impact on research: The paper shows that hierarchical, multi-stage verification can beat flat single-stage speculative decoding by exploiting the natural entropy gradient of model families that share tokenizers. This reframes multi-stage speculation as a practical, training-free option rather than one requiring specialized architectures or complex coordination.

Real-world applications:

• Interactive chat and assistant systems where low latency on consumer hardware matters — the paper reports 124 tokens per second on a single RTX 4090.
• Deployment on memory-constrained devices, where the paper's 1B draft / 8B target configuration trades a minimal amount of target quality for improved throughput.
• Production serving pipelines that prioritize predictable performance, where PSD_A's low variance across runs is described as well-suited to production scenarios.
• Accelerating inference of already-deployed model families (e.g., Llama 3.2/3.1) without retraining, since PyramidSD requires no additional training.

Industry relevance: The work comes from LG Electronics USA, and its reliance on off-the-shelf, tokenizer-compatible model families makes it directly applicable to existing inference pipelines rather than requiring custom model training.

Future Directions

• Adaptive controllers: Dynamically adjusting divergence thresholds and speculative lengths based on real-time decoding context, aiming to combine PSD_F's peak performance with PSD_A's stability.
• Generalization beyond homogeneous families: Enabling cross-family and mixed-attention setups would broaden applicability, particularly as emerging architectures diverge in tokenizers and scaling behaviors.
• Principled model selection: Understanding how model size ratios and entropy distributions influence acceptance rates could guide model selection and hyperparameter tuning.
• Taming hyperparameter complexity: The interaction between thresholds (τ_Q, τ_T) and speculative lengths (ℓ_D, ℓ_Q) varies across tasks, datasets, and model families, which the paper notes diminishes the plug-and-play capability of speculative decoding and motivates more principled algorithms.

Target Audience

This paper is most useful to machine learning systems engineers and LLM inference researchers who work on decoding efficiency and serving latency, as well as practitioners deploying open-weight model families under memory or latency constraints. Readers already familiar with speculative decoding will get the most from the threshold and speculative-length analysis; those new to the topic can still follow the conceptual framing of the three-model hierarchy.

Authors’ abstract

Speculative Decoding (SD) accelerates inference in large language models by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, the throughput gains of SD are fundamentally limited by a trade-off between draft model size and token acceptance: smaller draft models generate tokens more quickly but exhibit greater divergence from the target model, resulting in lower acceptance rates and reduced speedups. We introduce Pyramid Speculative Decoding (PyramidSD), an extension of SD that inserts an intermediate qualifier model between the draft and target to bridge the distributional gap in output predictions, allowing smaller model to be used for drafting. This hierarchical decoding strategy improves alignment across models, enabling higher acceptance rates and allowing the use of significantly smaller draft models without sacrificing overall performance. PyramidSD builds on fuzzy acceptance criteria to support relaxed divergence thresholds at each stage, improving throughput. In experiments, PyramidSD achieves up to 1.91x generation speed over standard SD, reaching 124 tokens per second on a consumer GPU (RTX 4090). In small-memory settings with a 1B-parameter draft model and an 8B target model, PyramidSD minimally trades target model quality for improved throughput. Overall, PyramidSD offers a practical approach to enhancing speculative decoding efficiency and can be readily applied to existing inference pipelines.