AdaFuse: Adaptive Ensemble Decoding with Test-Time Scaling for LLMs

AI summary

Overview

Research area: Natural language processing — inference-time ensembling and decoding strategies for large language models.

Technical level: Advanced. The paper assumes familiarity with autoregressive decoding, tokenization, beam search, log-likelihood scoring, and the sample/span/token-level taxonomy of LLM ensembling.

Scope: A decoding framework that ensembles multiple open-source LLMs at word-level granularity, using model confidence to decide when to fuse and when to explore alternative continuations during generation.

What This Paper Is About

Different LLMs have complementary strengths because of differences in pretraining data, architecture, and decoding behavior, but combining them usually requires either retraining or fixed rules about when and how to merge their outputs. Existing ensemble methods fuse at a fixed granularity — whole responses, fixed spans, or individual tokens — and therefore cannot adapt mid-generation to how uncertain the models actually are. AdaFuse addresses this by fusing model outputs at word boundaries and letting a confidence signal decide, step by step, whether to keep generating directly or pause and re-score competing candidates.

Key Contributions

1. Confidence-guided adaptive decoding. AdaFuse uses the probability margin between the top-1 and top-2 first tokens at the start of each word as a proxy for confidence, then decides whether to commit to a longer multi-word span or stop and re-score. This replaces fixed-length spans with dynamic, uncertainty-driven commitment.

2. Diversity-aware ensemble scaling. When confidence is low, AdaFuse triggers a two-stage word-level search that separates exploration (selecting the top-B distinct initial tokens) from exploitation (greedily extending each into a complete word). Diversity is applied only in uncertain states, so compute is not spent uniformly across the generation.

3. Word-level cross-model scoring. Candidate word spans from all models are pooled into a union set and scored with a normalized negative log-likelihood (NLL) averaged across models. Word boundaries serve as a tokenizer-agnostic alignment unit, avoiding the vocabulary alignment problem that constrains token-level ensembling.

4. Empirical validation across task families. AdaFuse outperforms four strong ensemble baselines on open-domain QA, reading comprehension, arithmetic reasoning, and bidirectional machine translation, with an average relative improvement of 6.88% under a matched computational budget.

Main Findings

• Consistent average gains over the best baseline. AdaFuse (fixed base pair: Mistral-7B-Instruct-v0.3 + LLaMA-3.1-8B-Instruct) reaches an average score of 63.23 across six benchmarks versus 59.16 for the strongest baseline, SweetSpan — a 4.07-point absolute and 6.88% relative improvement.

• Largest relative gains on GSM8K and Natural Questions. Relative to the best baseline, AdaFuse improves by 17.03% on GSM8K, 10.01% on Natural Questions, 6.04% on Flores En→De, 5.60% on Flores De→En, 4.12% on SQuAD, and 0.97% on TriviaQA.

• Base-model compatibility matters more than ensembling alone. With the fixed pair, AdaFuse scores 79.15 on GSM8K, below LLaMA-3.1-8B-Instruct's 81.05, because the two base models differ sharply in arithmetic ability. Switching to the top-2 base pair (Qwen3-8B + LLaMA-3.1-8B-Instruct) raises GSM8K to 90.25, showing that pairing models of comparable strength on a task is decisive.

• Adaptive word commitment beats fixed-length commitment. Ablations with fixed word lengths of 1, 2, and 3 all underperform the adaptive rule. A qualitative Natural Questions example shows fixed-length decoding converging on the wrong answer ("Terrence"), while adaptive commitment allows one model to produce the correct multi-word entity ("Kelly Reno") within a single round.

• Commitment length tracks task type. The distribution of words generated per decoding round differs systematically: Natural Questions is [80.29, 15.53, 4.18]% for 1/2/3 words; SQuAD is [57.64, 21.27, 21.09]%; and German→English translation shifts to [34.02, 19.89, 46.09]%, indicating the decoder adapts span length to the task rather than enforcing a constant.

• Diversity-aware scaling gives a reliable boost. Increasing the branching factor B from 1 to 5 shows a generally upward accuracy trend on NQ, TriviaQA, and GSM8K. At B = 3, enabling diversity-aware scaling improves five of six benchmarks; a beam-search variant that ignores diversity degrades as candidates increase.

• Favorable runtime profile. On NQ with 4×A100 80GB GPUs, AdaFuse runs comparably to UniTE (a lightweight token-level method), is substantially faster than the span-level SweetSpan, and outperforms vocabulary-alignment-heavy methods such as DeepEn in wall-clock time.

• The confidence threshold is robust in a broad region. Sensitivity analysis shows degraded performance at extreme values of τ_Δ, but stable accuracy around the chosen setting of 0.7.

Methodology in Plain English

AdaFuse runs several LLMs side by side and merges their outputs during generation rather than after it finishes.

The core decoding loop works like this:

1. Propose word candidates. From the current prefix, each model extends generation token by token until it reaches a whitespace, producing a complete word rather than a fragment. Word boundaries give a shared unit that works even when models use different tokenizers.

2. Decide whether to trust or pause. At the start of each new word, AdaFuse looks at how far apart the model's top two next-token probabilities are. A large margin means the model is confident, so it continues greedily for up to three words in a row. A small margin — below the threshold of 0.7 — signals uncertainty, so generation stops and the system re-scores.

3. Explore alternatives only when uncertain. Under uncertainty, AdaFuse runs a two-stage search: it picks the top B distinct first tokens (exploration), then greedily completes each one into a full word (exploitation). This avoids the redundancy that plain beam search produces at word level, since beam search optimizes token probability rather than lexical diversity.

4. Score candidates jointly across models. All candidate spans from all models go into a shared pool. Each span gets a normalized NLL — roughly, how surprised each model is by the span's tokens, averaged and length-normalized. These scores are averaged across models to produce a fusion score, and the span with the lowest score wins.

5. Append and repeat. The winning span is concatenated to the prefix and the loop restarts until an end-of-sequence token or length limit is reached.

The design creates a feedback loop between the two mechanisms: ensemble decisions direct where exploration should happen, and the diversity generated during exploration improves the quality of subsequent ensemble decisions.

Experimental setup: Four open-source models (LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen3-8B, InternLM3-8B-Instruct) were evaluated against four baselines (LLM-Blender, DeepEn, SweetSpan, UniTE) on six benchmarks: Natural Questions, SQuAD, TriviaQA, GSM8K, and Flores En→De and De→En. Total compute was roughly 500 A100 GPU-hours.

Why This Matters

Impact on research. The paper reframes ensemble granularity as a variable to be controlled rather than a fixed design choice, and it demonstrates that ensembling and test-time scaling are mutually reinforcing rather than independent techniques. That framing opens a research direction around jointly optimizing inference-time compute allocation and multi-model fusion.

Real-world applications:

• Open-source model deployment. Organizations serving open-weight models can combine several mid-size checkpoints to approach the quality of a single larger closed model without retraining or additional pretraining cost.

• Multilingual translation services. AdaFuse shows gains on both En→De and De→En, and its word-boundary alignment avoids tokenizer mismatch — useful where models trained on different tokenizers must be combined.

• Knowledge-intensive QA systems. The largest absolute gains appear on open-domain and reading-comprehension tasks, where individual models frequently hallucinate or drift from evidence.

• Cost-constrained inference. Because exploration only activates under uncertainty, AdaFuse spends extra compute where it is likely to change the answer, which matters for deployments with fixed GPU budgets.

Industry relevance. The method requires token-level likelihoods and tokenizer access, which rules out black-box commercial APIs but fits naturally with self-hosted open-weight stacks. The runtime results suggest AdaFuse is practical at production latency, not just in research settings.

Future Directions

• Extending to black-box APIs. The stated limitation is that AdaFuse needs token-level likelihoods and tokenizer outputs. Adapting the scoring to models accessible only through text-level APIs — for instance via sampled-output statistics or preference proxies — remains unsolved.

• Automatic base-model selection. The GSM8K results show that ensembling a strong and a weak model on a given task can hurt. A method for predicting which model combinations will be complementary before running inference would make the framework far more usable.

• Learned rather than fixed thresholds. τ_Δ is set globally to 0.7 and M to 3. Making the confidence criterion and commitment cap adaptive per task, per model, or per decoding position could yield further gains.

• Energy and throughput accounting. The paper notes that extra forward passes increase GPU energy use but argues that fewer decoding rounds offset it. A systematic measurement of energy, latency, and quality trade-offs across ensemble sizes (the appendix reports accuracy gains from two to four models) would clarify whether this holds broadly.

Target Audience

Researchers and practitioners working on LLM inference, decoding strategies, and multi-model systems. The paper is most valuable to those with a working understanding of autoregressive decoding and tokenization, particularly engineers deploying open-weight model ensembles under latency or cost constraints, and researchers interested in test-time compute scaling, model fusion, or uncertainty-aware generation. Readers seeking an introduction to LLM ensembling will find the related-work taxonomy of sample-, span-, and token-level methods useful as a map, but the methodology sections assume substantial background.

Authors’ abstract

Large language models (LLMs) exhibit complementary strengths arising from differences in pretraining data, model architectures, and decoding behaviors. Inference-time ensembling provides a practical way to combine these capabilities without retraining. However, existing ensemble approaches suffer from fundamental limitations. Most rely on fixed fusion granularity, which lacks the flexibility required for mid-generation adaptation and fails to adapt to different generation characteristics across tasks. To address these challenges, we propose AdaFuse, an adaptive ensemble decoding framework that dynamically selects semantically appropriate fusion units during generation. Rather than committing to a fixed granularity, AdaFuse adjusts fusion behavior on the fly based on the decoding context, with words serving as basic building blocks for alignment. To be specific, we introduce an uncertainty-based criterion to decide whether to apply ensembling at each decoding step. Under confident decoding states, the model continues generation directly. In less certain states, AdaFuse invokes a diversity-aware scaling strategy to explore alternative candidate continuations and inform ensemble decisions. This design establishes a synergistic interaction between adaptive ensembling and test-time scaling, where ensemble decisions guide targeted exploration, and the resulting diversity in turn strengthens ensemble quality. Experiments on open-domain question answering, arithmetic reasoning, and machine translation demonstrate that AdaFuse consistently outperforms strong ensemble baselines, achieving an average relative improvement of 6.88%. The code is available at https://github.com/CCM0111/AdaFuse.