When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling

AI summary

Overview

• Research area: Natural Language Processing — inference-time ensembling of Large Language Models (combining next-token probability distributions from multiple models), with a focus on long-form / Chain-of-Thought generation.
• Technical level: Advanced. The paper assumes familiarity with tokenization schemes, next-token probability distributions, autoregressive decoding, speculative decoding, and KV caching.
• Scope in one sentence: The paper argues that probability-level LLM ensembling should be applied selectively — only at token positions where it is both safe and necessary — and proposes SAFE, a generate–verify–ensemble framework that decides those positions using tokenization mismatch and model agreement.

What This Paper Is About

Probability-level ensembling works by averaging the next-token probability distributions of several LLMs and picking the most confident token, and it has worked well for short answers and multiple-choice questions. The authors show this practice breaks down in long-form generation, because ensembling at every token introduces tokens that conflict with a participating model's tokenizer and corrupts its probability distribution, while also being expensive. The goal is to decide when to ensemble so that long-form Chain-of-Thought generation becomes both stable and fast.

Key Contributions

1. A diagnosis of why ensembling fails in long-form generation. The authors identify two factors governing where ensembling should happen: tokenization mismatch across models (an issue of stability) and consensus in models' next-token probability distributions (an issue of efficiency). They introduce the notion of "OOV-like" tokens — tokens that are not truly out-of-vocabulary but that force a model to condition on an invalid prefix.
2. SAFE (Stable And Fast LLM Ensembling). A generate–verify–ensemble cycle in which one model acts as a drafter producing a lookahead sequence of tokens, and the remaining models act as verifiers that examine those tokens in a single forward pass and flag which ones require ensembling. Ensembling is triggered only when (i) no OOV-like token is introduced and (ii) the verifiers show insufficient agreement.
3. Probability sharpening. Two strategies to fix an overly smooth ensemble distribution (where max P_ens < 0.5) caused by probability mass being scattered across sub-word tokens: a heuristic that reallocates mass from variant subword tokens to their common prefix token above a threshold λ, and a geometric-mean alternative to the arithmetic mean.
4. A plug-and-play method plus KV cache management. SAFE is applied on top of the existing GaC and UniTE ensemble methods by adding generate–verify logic. The authors also implement KV cache updating at the end of every ensemble step to keep cache and sequence consistent, which prior work left as future work, and apply it to all baselines.

Main Findings

• Ensembling every token degrades Chain-of-Thought performance. UniTE, which ensembles at every generation step, underperforms individual models across all CoT experiments: on MATH500 it reaches 59.6 (-15.2 relative to the best individual model), on MMLU-redux 73.39 (-3.50), and on ARC-C 87.97 (-2.30). By contrast, without CoT it matches or beats the individual models on multiple-choice questions (69.36 on MMLU-redux, 88.40 on ARC-C), since tokenizer mismatches do not arise there.
• SAFE restores ensemble gains with few ensemble operations. With two-model ensembling (Internlm3 + Qwen2.5), UniTE + SAFE reaches 77.81 on MMLU-redux, 77.4 on MATH500, 92.04 on GSM8K, 82.97 on BBH, and 90.78 on ARC-C, for an average of 84.20 (+1.33), while ensembling fewer than 20% of tokens. UniTE+SAFE is reported as best in 9/15 cases, and some configurations ensemble fewer than 1% of tokens (for example E/T of 0.67, 0.71, 0.72, and 0.82 in various cells).
• Math needs far less ensembling than general domains. With UniTE + SAFE, ensembling is triggered for only 4.85% of tokens on average in math datasets, versus 15.24% in general-domain datasets — nearly three times higher. The authors attribute this to equations and structured expressions having limited variation, which raises agreement among verifiers.
• More models is not always better. Ensembling three models did not consistently beat two-model ensembling (three-model UniTE + SAFE: 84.59 average). Ensembling the top-2 best-performing models typically gave the strongest results; when rankings are unknown, ensembling several comparable models gives stable but not necessarily optimal performance.
• SAFE also helps when tokenizers are nearly identical. On Qwen2-7B + Llama-3.1-8B, where more than 99% of Oxford 5000 words are tokenized identically, GaC + SAFE improves MATH500 from 52.4 to 59.4 (+9.6 over the best individual model's 49.8), and UniTE + SAFE improves it to 55.6.
• Probability sharpening consistently helps. In the ablation with Internlm3 + Qwen2.5, UniTE + SAFE without sharpening scores 77.53 / 76.6 / 91.66 (MMLU-redux / MATH500 / GSM8K, average 81.93), versus 77.81 / 77.4 / 92.04 (average 82.42) with the heuristic at λ = 0.1. The geometric mean scored highest in that comparison at 78.31 / 77.6 / 92.27 (average 82.73).
• A drafter chunk of 5 tokens is the best balance. MMLU-redux / GSM8K / ARC-C results are 77.67 / 91.66 / 90.19 for n = 3, 77.81 / 92.04 / 90.78 for n = 5, and 78.31 / 92.04 / 90.78 for n = 8; longer sequences do not hurt accuracy but may reduce efficiency.
• Latency approaches that of a single model. The authors report that SAFE closely matches individual-model latency when generating hundreds of tokens, regardless of the underlying ensemble method, attributing this to three properties: only the drafter does autoregressive generation, far fewer ensemble operations, and the KV caching strategy.
• Tokenization diversity motivates the setup. Across the three main models (Internlm3-8B-Instruct, Qwen2.5-7B-Instruct, EXAONE-3.5-7.8B-Instruct), agreement rates on the Oxford 5000 words range only from 40% to 60%; for Qwen2-7B-Instruct and Llama-3.1-8B-Instruct, more than 99% of those words are tokenized identically.

Methodology in Plain English

The authors split the participating models into two roles. One model — chosen as the best-performing one — is the drafter, and it generates a chunk of tokens (5 by default) ahead of time. The other models are verifiers.

For each chunk, the verifiers do a single forward pass and check the drafter's tokens in order. They stop at the first token that satisfies two conditions. First, the token immediately before it must not be an "OOV-like" token — meaning the drafter's token boundary cannot conflict with the verifiers' tokenization boundaries. For example, if the drafter produces the word "Incorrect" as three tokens (Inc, orr, ect) but another model uses "Incorrect" as one token, then "Inc" and "orr" are OOV-like and ensembling is skipped there, while the boundary after "ect" is safe. Second, the verifiers must not already agree strongly on the token — measured by either unanimous agreement across verifiers, or an average probability above one half across all models, which the authors prove in an appendix does not compromise accuracy versus using the exact ensemble distribution.

Only when both checks pass is the ensemble distribution actually built (averaging the models' distributions) and the most confident token substituted. If that ensemble distribution is too smooth (max probability below 0.5), a sharpening step concentrates the mass — either by moving probability from variant subword tokens to their common prefix (only for tokens above threshold λ = 0.1 in the main experiments), or by using a geometric mean instead of an arithmetic mean. The drafter then resumes generation from the chosen token, and the cycle repeats.

Experiments use five benchmarks under zero-shot CoT (BBH uses 3-shot CoT): MMLU-redux (30 subjects), MATH500, GSM8K, ARC-Challenge, and BBH (15 selected subjects). Models include Internlm3-8B-Instruct, Qwen2.5-7B-Instruct, EXAONE-3.5-7.8B-Instruct, Qwen2-7B-Instruct, and Llama-3.1-8B-Instruct; 32B-scale models are explored in an appendix. All models use greedy decoding with a maximum output length of 2048, each model is loaded on a separate RTX 3090 GPU with FP16 precision and FlashAttention-2 enabled.

Why This Matters

The paper reframes LLM ensembling as a scheduling problem rather than a pure aggregation problem: when you combine models is as important as how. It shows that a technique widely reported as effective for short answers silently harms long-form reasoning, and that a large share of ensemble operations can be skipped without losing accuracy.

Real-world applications:

• Deploying multi-model reasoning assistants where latency and GPU budget matter, since SAFE is reported to approach single-model latency while retaining ensemble-level accuracy.
• Cost-controlled serving of Chain-of-Thought math and reasoning systems, where the authors report only 4.85% of tokens need ensembling on average.
• Combining models from different vendors or families whose tokenizers differ, where naive ensembling currently fails.
• Retrofitting existing ensembling pipelines, since SAFE is described as plug-and-play on top of GaC and UniTE.

Industry relevance: the work targets a practical bottleneck for anyone serving more than one LLM — the cost of aligning large heterogeneous vocabularies and running autoregressive generation on every model — and offers a route to ensemble-level quality at close to single-model cost.

Future Directions

• Choosing the sharpening strategy adaptively, since the geometric mean performed best in the reported comparison while the heuristic form remains useful where an arithmetic mean is required; the authors explicitly leave this as a flexible design decision.
• Extending the approach beyond the tested 7B-scale models (32B-scale experiments are only sketched in an appendix) and understanding how the results scale.
• Characterizing precisely when ensembling more than two models pays off, since three-model ensembling did not consistently beat two-model ensembling and optimal model ranking is assumed to be known in the strongest configurations.
• Tailoring ensembling frequency by domain, given the large gap between the 4.85% ensemble rate on math datasets and 15.24% on general-domain datasets.

Target Audience

Researchers and engineers working on inference-time model combination, LLM serving systems, and efficient decoding — particularly those already familiar with speculative decoding or probability-level ensembling such as GaC and UniTE. It also suits practitioners who must ensemble models with different tokenizers and need to know where the technique is safe to apply.

Authors’ abstract

Ensembling Large Language Models (LLMs) has gained attention as a promising approach to surpass the performance of individual models by leveraging their complementary strengths. In particular, aggregating models' next-token probability distributions to select the next token has been shown to be effective in various tasks. However, while successful for short-form answers, its application to long-form generation remains underexplored. In this paper, we show that using existing ensemble methods in long-form generation requires a careful choice of ensembling positions, since the standard practice of ensembling at every token often degrades performance. We identify two key factors for determining the ensembling positions: tokenization mismatch across models and consensus in their next-token probability distributions. Based on this, we propose SAFE, (Stable And Fast LLM Ensembling), a framework that selectively ensembles by jointly considering these factors. To further improve stability, we apply a probability sharpening strategy when the ensemble distribution becomes overly smooth, enabling the selection of more confident tokens during ensembling. Our experiments on diverse benchmarks, including MATH500 and BBH, demonstrate that SAFE outperforms existing methods in both accuracy and efficiency, with gains achieved even when ensembling fewer than 1% of tokens.