ProVe: Proposer-Verifier Decoding

AI summary

Overview

Research area: Natural Language Processing — specifically efficient inference and decoding for large autoregressive (AR) language models.

Technical level: Advanced. The paper assumes familiarity with autoregressive decoding, speculative decoding, draft/verify pipelines, and fine-tuning of large checkpoints.

Scope: The paper proposes ProVe, a decoding scheme that inverts standard speculative decoding by turning a large model into a block-level proposer and using a verifier (itself or a smaller model) to check whole blocks of tokens in a single forward pass.

What This Paper Is About

Standard speculative decoding speeds up text generation by having a small model draft tokens one at a time while a large model checks them in parallel. This paper asks what happens if those roles are reversed: the large model is fine-tuned to propose several future tokens at once, and a separate forward pass verifies the proposal. The goal is faster generation from a standard checkpoint without training a dedicated draft model from scratch, while offering a tunable tradeoff between speed and accuracy.

Key Contributions

1. Role inversion for speculative decoding. The paper flips the conventional draft-then-verify setup so that the large model proposes a block of future tokens in a single forward pass, and verification costs one additional forward pass.

2. Reusing a standard AR checkpoint as the proposer. Rather than training a separate draft head or a standalone small draft model, the authors fine-tune an existing autoregressive checkpoint into a block proposer.

3. Two operating modes. ProVe-lossless uses the proposer itself as the verifier and is reported to reproduce the proposer's greedy AR decoding. ProVe uses a smaller AR model as the verifier, which yields a controllable speed-versus-accuracy tradeoff.

4. Evaluation across six benchmarks using a 32B proposer and, in the tradeoff setting, a 3B verifier.

Main Findings

• Block proposals leave real headroom. With a block size of 16, the proposed block has an average AR-matching prefix of 6.6 tokens on GSM8K, meaning a substantial portion of each proposal already agrees with what the model would have generated normally.

• Lossless mode is substantially faster. ProVe-lossless with a 32B proposer reaches 2.5x average AR throughput across six benchmarks, without training a separate draft head.

• Small verifier trades accuracy for more speed. ProVe with a 3B verifier reaches 2.7x average throughput while sitting 2.6 accuracy points below AR on the six-task average.

• The speed/accuracy balance is selectable. Because the verifier can be the proposer itself or a smaller model, the choice of verifier acts as the knob controlling the tradeoff.

• Details not in the abstract. The abstract does not report per-benchmark results, the composition of the six benchmarks, training data or compute for the proposer fine-tuning, or comparisons against other speculative decoding baselines.

Methodology in Plain English

The authors begin with an ordinary autoregressive checkpoint — a model that normally writes one token at a time. They fine-tune it so that, in a single forward pass, it emits a whole block of upcoming tokens at once. That fine-tuned model becomes the proposer.

A second forward pass then checks the proposed block. How that check is performed determines the mode. In ProVe-lossless, the proposer itself does the checking, and the authors state this reproduces the model's own greedy autoregressive output — so nothing is lost relative to normal decoding. In plain ProVe, a smaller autoregressive model performs the check, which is cheaper but may accept or reject proposals differently, introducing a small accuracy gap.

The core intuition is that block proposals can already match the model's normal output for a good number of leading tokens (6.6 out of a 16-token block on GSM8K), so much of the sequential decoding work can be replaced by parallel proposal plus verification.

Why This Matters

Research impact. The work challenges a common assumption in speculative decoding — that the drafter must be a separate, smaller model. Showing that the large model itself can be repurposed as a block proposer through fine-tuning suggests a different axis for inference acceleration research: adapting the target model rather than adding auxiliary models.

Real-world applications.

• Latency-sensitive interactive systems such as chat assistants and coding copilots, where per-token delay is visible to users.
• Serving pipelines for long-form generation, where throughput gains compound over many output tokens.
• Deployments that want a small, cheap verifier (the 3B setting) alongside a much larger proposer, giving operators a dial for cost versus quality.
• Settings where teams already have a large checkpoint and want speedups without maintaining a second trained draft model.

Industry relevance. Avoiding a separately trained draft head simplifies the model inventory and training pipeline for serving infrastructure. An explicit, measurable tradeoff between throughput and accuracy is also directly useful for capacity planning and for setting service-level agreements.

Future Directions

• How far can block proposal accuracy go? The 6.6-token average matching prefix out of 16 leaves clear room; better proposer training objectives or block sizes could shift this.
• Does fine-tuning into a proposer affect general model quality? The abstract does not characterize any change in the model's ordinary capabilities after conversion, which is an open question for adoption.
• Adaptive verifier selection. Choosing among verifier sizes per task or per accuracy budget — rather than a fixed 3B verifier — is a natural extension of the reported tradeoff.
• Generalization beyond the six reported benchmarks. Whether the throughput gains and the accuracy gap hold on other task types, model families, and scales is not established in the abstract.

Target Audience

Researchers and engineers working on LLM inference efficiency, speculative decoding, and serving systems, particularly those who already understand standard draft-and-verify pipelines. It is also relevant to practitioners deciding whether to add a separate draft model to a production stack, and to graduate-level readers studying decoding-time acceleration. Beginners will need background on autoregressive and speculative decoding before the contribution is clear.

Authors’ abstract

Speculative decoding accelerates autoregressive (AR) language models by letting a small model draft tokens serially and a large model verify them in parallel. We reverse these roles. Starting from a standard AR checkpoint, we fine-tune the large model into a proposer that predicts a block of future tokens in a single forward pass, then use one additional forward pass to verify the proposed block. This gives ProVe, Proposer-Verifier decoding. The verifier can be a smaller AR model or the proposer itself; the latter gives ProVe-lossless, which reproduces the proposer's greedy AR decoding. Block proposals already provide substantial room for acceleration: with a block size of 16, the proposed block has an average AR-matching prefix of 6.6 tokens on GSM8K. Across six benchmarks with a 32B proposer, ProVe-lossless reaches 2.5x average AR throughput without training a separate draft head. With a small 3B verifier, ProVe reaches 2.7x average throughput at 2.6 accuracy points below AR on the six-task average, providing a controllable tradeoff between speed and accuracy.