Skip to content
AI.info

The Pulse

NVIDIA posts draft models for Qwen3.8-27B decoding

NVIDIA has published three external draft models for speculative decoding with Qwen3.8-27B. Its largest draft averaged 5.58 tokens per target step on a qualitative benchmark, compared with 3.81 for the target model’s built-in MTP head.

NVIDIA posts draft models for Qwen3.8-27B decoding

NVIDIA has published three external draft models designed to speed up text generation with Qwen3.8-27B through speculative decoding. The largest draft has 4.21 billion parameters and is available on Hugging Face under the NVIDIA Open Model Agreement.

In NVIDIA’s evaluation, that model returned an average of 5.58 tokens per target step on the qualitative subset of SPEED-Bench, compared with 3.81 for Qwen3.8-27B’s built-in multi-token prediction, or MTP, head. The test used a draft block size of 16.

How the draft works

The largest draft is based on Qwen3.5-4B and is a complete language model, rather than a draft head. It shares the target model’s tokenizer and vocabulary and proposes blocks of candidate tokens for Qwen3.8-27B to verify.

It was trained with NVIDIA Model Optimizer in external speculative decoding mode. The draft samples candidates under the target’s serving policy, using a temperature of 1.0, top-k of 20 and top-p of 0.95; the target then checks them by rejection sampling.

NVIDIA says the accepted sequence has exactly the distribution the target would have produced on its own. The final output text comes from Qwen3.8-27B.

Benchmark comparison

On SPEED-Bench’s long-context throughput-32k subset, the 4.21-billion-parameter draft averaged 5.11 tokens per target step, against 3.86 for the built-in MTP head, according to the model card. That evaluation also used a draft block size of 16.

The qualitative subset contains 1,138 conversation turns across categories including coding, math, multilingual tasks and writing. The long-context subset contains 1,536 single-turn prompts of 33k to 46k tokens, divided evenly among low-entropy, mixed and high-entropy categories.

In the same comparison, the Qwen3.5-0.8B draft has 0.75 billion parameters and averaged 3.92 on the qualitative subset and 3.60 on throughput-32k. The Llama-3.2-1B draft has 1.48 billion parameters and averaged 3.42 and 3.07, respectively.

All three external drafts were trained on the same corpus against the same target and evaluated with the same harness, according to NVIDIA. Their scores are averages across categories with a draft block size of 16.

Training and deployment

The training set comprised 199,998 single-turn conversations. Prompts came from the Nemotron-Post-Training-Dataset-v2, while Qwen3.8-27B synthesized the responses; prompts were sampled in equal shares across nine categories, including STEM, code, math, chat and five languages.

Prompts longer than 3,072 tokens were dropped. The target’s top-20 output distribution at each response position was recorded as the training signal, and the largest draft was trained at a sequence length of 12,288 tokens.

The intended deployment for the drafts is NVIDIA Groq 3 LPU hardware, where they are compiled for its runtime. The published checkpoints are standard Hugging Face models and also load in vLLM, SGLang and TRT-LLM.

The largest draft is available for commercial or non-commercial use under the NVIDIA Open Model Agreement. The model card says its base model’s training data contains toxic language and societal biases and that generated answers may be inaccurate, omit information or include irrelevant or undesirable text.

Source

  1. NVIDIA

Related stories

Explore

More articles