One Temperature to Rule Them All?

AI summary

Overview

Research area: Inference-time compute for large language models — specifically, how sampling temperature should be chosen when a model is queried multiple times per problem (the pass@K setting).

Technical level: Intermediate to Advanced. The core argument is intuitive, but the paper leans on convex optimization, concentration bounds, and first-order asymptotic analysis to prove its claims.

Scope (one sentence): The paper asks whether using multiple different temperatures across K samples actually beats a single well-tuned temperature, and answers with both an extensive empirical study (four models, three benchmarks) and a theoretical explanation.

What This Paper Is About

When an LLM is sampled several times for the same question, temperature controls the tradeoff between reliability (low temperature) and diversity (high temperature). Recent work has suggested that different questions have different ideal temperatures, so spreading samples across a range of temperatures should beat committing to one. In practice, however, single-temperature sampling remains the standard. This paper investigates that gap in the canonical pass@K setting: it characterizes how per-question success probability varies with temperature, derives an efficient approximation to the theoretically optimal multi-temperature allocation, and explains — via two "dulling effects" — why the gains from temperature diversity turn out to be modest once temperature is tuned as a function of the sampling budget K.

Key Contributions

1. A characterization of question "temperature profiles." The authors show that the probability of solving a question as a function of temperature has a simple, essentially unimodal shape, and that questions split into two canonical types: those whose optimum is at τ = 0, and those that rise to a positive optimum and then decay. They argue these are the only possible qualitative forms given the boundary conditions P(0) ∈ {0,1} and P(τ) → 0 as τ → ∞, together with unimodality.

2. A hardness-based analysis of temperature behavior. Defining hardness as the effective number of samples needed for a reliable solve (H_q(τ) = -1/ln(1-P_q(τ)), targeting success probability 1 - 1/e), they show the average optimal temperature varies with hardness, while both the size of each question's efficient temperature range and the overlap of those ranges across questions shrink — but only logarithmically — as hardness grows.

3. An efficient method for the optimal multi-temperature allocation. They discretize temperatures, reframe the problem as allocating integer sample counts K_1, …, K_l across a fixed temperature set subject to K_1 + ⋯ + K_l = K, and show (Theorems B.1 and B.2) that the expectation over the question distribution can be approximated with a relatively small randomly sampled subset of questions. Relaxing integrality makes the problem convex and efficiently solvable; rounding recovers a good integer solution.

4. Two "dulling effects" explaining why multi-temperature gains are small. The Short-Sampling Effect (splitting samples across suboptimal temperatures costs exponentially, whereas using one compromise temperature costs only linearly) and the Goldilocks Sampling Effect (only questions whose hardness is on the scale of K meaningfully distinguish sampling strategies).

Main Findings

• Profiles are simple and near-universal. Across all models and datasets, only a negligible fraction of questions deviated from the two canonical profile shapes, and most deviations were not clearly distinguishable from statistical uncertainty.

• Optimal temperature shifts with hardness. The average optimal temperature varies significantly across hardness levels, not just across individual questions. The paper distinguishes these two sources of variation because they have different implications for strategy design.

• Separation shrinks only logarithmically. Both the efficient range size (temperatures achieving at least half the optimal success probability) and the maximal overlap ratio R_Q(H) decay logarithmically with hardness. The overlap decay is less pronounced than the range-size decay, and for some models a cluster of easy questions with near-zero optimal temperature obscures the trend at small K.

• Optimal allocation wins, but modestly. The optimal allocation has about a 10% advantage over each fixed single-temperature strategy for at least some values of K. But once temperature is allowed to vary as a function of K, the advantage becomes much more limited — for each K, at least one single-temperature scheme achieves comparable performance.

• Uniform temperature sampling is a reasonable fallback. Although it underperforms finely tuned single-temperature strategies, the uniform scheme of [27] never deviates substantially, because it always allocates some samples to the appropriate temperature region.

• The Short-Sampling Effect. In the simplified two-question setting, the two-expert strategy beats the single compromise temperature if and only if P_E > 2P_C (to first order). The asymmetry matters: over-splitting resources across suboptimal temperatures incurs exponential loss, while under-splitting (using one compromise temperature) incurs only linear loss — the authors compare this to short selling in financial markets. The condition P_C ≈ P_E / 2 motivates their definition of the efficient range.

• The Goldilocks Sampling Effect. The marginal gain in success probability from an extra sample is maximized exactly when K = -1/ln(P̄) = H. If the sampling scale is off by a factor α (K = αH), the normalized marginal gain is α e^{1-α}. This stays at or above 1/e only for roughly 0.16 ≲ α ≲ 3.15, an efficient range spanning about a factor of 20 in hardness — a relatively well-localized regime given the variability of question difficulty.

• Practical robustness to distribution mismatch. Because only near-K-hardness questions matter, temperature choices transfer across hardness distributions. In reweighting experiments, even when the KL divergence exceeds 3, on average roughly 90% of reweightings lead to less than a 10% performance loss; for KL divergences between 1 and 2, that fraction rises to 99%. The required reweighting to make a 10%-worse temperature win grows approximately exponentially with the hardness-window width.

• Hardness need not be known. A practical consequence: temperatures can be selected knowing only the computational budget K, not individual question hardness, and without closely matching the tuning and test difficulty distributions.

Methodology in Plain English

Empirical side. The authors take four models — Qwen3-0.6B, Qwen3-8B, Mistral 7B Instruct, and GPT-OSS 20B — and three benchmarks: LiveCodeBench (V1–V6) for code generation, MATH-500 for mathematics, and Hi-ToM for multiple-choice higher-order theory of mind. For each model and question, they draw typically 1,500 samples at each of 16 to 38 temperatures between 0.075 and 4.2, grading automatically by unit tests, final answers, and chosen options respectively. From these samples they estimate, for every individual question, a curve of "probability of solving it" versus temperature. Considerable care went into the statistics (Appendix D) because very hard questions with only a handful of successful samples can badly distort these curves.

Turning curves into a decision problem. For K independent samples at temperatures τ_1, …, τ_K, the failure probability is simply the product of the per-sample failure probabilities. So to find the best multi-temperature scheme you minimize an average of products over the question distribution. Directly optimizing over a continuous temperature range is intractable, so they restrict to a small fixed set of candidate temperatures and instead decide how many of the K samples go to each. Relaxing "how many" from integers to real numbers makes the problem convex and solvable; standard rounding arguments then recover a near-optimal integer allocation. A concentration argument shows a modest random subset of questions suffices to approximate the expectation over the full distribution.

Explaining the gap. The empirical separation between questions (10%–20%) is far larger than the measured gain from multi-temperature sampling, so something must be eating the difference. The authors derive two mathematical mechanisms to account for it, proving a clean result in a simplified two-question Gaussian-like setting (Proposition 4.1) and then extending the intuition to multiple questions, asymmetric profiles, and non-vanishing tails (Propositions B.1–B.3). The Goldilocks result is a short calculus exercise but carries most of the explanatory weight.

Validation of predictions. They test the robustness prediction by randomly reweighting questions according to hardness, optimizing temperatures under the reweighted distribution, and evaluating on the original — measuring performance loss against KL divergence.

Why This Matters

Impact on research. The paper reframes a question that has been treated as an engineering detail — how to spend temperature across samples — as something with clean structure and provable limits. It gives the field an efficient, general-purpose approximation to the optimal allocation scheme (applicable to other hyperparameters such as min_p and top_h, per Appendix I), and it reconciles theory with practice: prior work suggested multi-temperature sampling should help a lot, and this paper explains why it does not. The two dulling effects are general enough to apply to almost any inference-time sampling hyperparameter, not just temperature.

Real-world applications:

• Code generation with unit-test verification — the LiveCodeBench setting analyzed here, where pass@K is directly the deployment metric and a few percentage points of pass rate translate to real economic value.
• Mathematical reasoning pipelines that sample many candidate solutions and select among them.
• Budget-aware inference serving — choosing a single temperature per K (rather than a per-question policy) is trivial to deploy and captures most of the benefit.
• Cheap dataset curation for temperature tuning — because temperature choices transfer across skewed hardness distributions, tuning data need not match deployment difficulty, reducing the cost of building the tuning set.

Industry relevance. Serving systems must pick sampling parameters before knowing whether a given question is easy or hard. The Goldilocks result says the decisive variable is the compute budget K, not the question — a deployable rule. And the asymmetry in the Short-Sampling Effect means the cost of being conservative (one temperature) is small while the cost of being aggressive (splitting samples across many temperatures) can be exponential, which favors the simpler, cheaper default.

Future Directions

1. Extending beyond pass@K. The analysis assumes independent samples with a binary success criterion. Multi-sampling pipelines with chain-of-thought, learned verifiers, or aggregation/selection steps are not covered, and the authors flag this as an important open direction.

2. Combining with adaptive strategies. Because the results say temperature can be selected from K alone without knowing question hardness, the authors suggest the pre-committed schemes studied here could be paired with adaptive, input-dependent temperature selection — where the hardness-independence property may be more valuable.

3. Higher-dimensional and multi-hyperparameter strategies. The authors conjecture that greater separation between questions may emerge as the dimensionality of the sampling configuration increases (e.g., jointly varying temperature with min_p and top_h), potentially enhancing the practical value of their allocation approximation.

4. Broadening empirical coverage and closing the theory gap. The appendices acknowledge that the space of architectures, training procedures, and tasks is enormous; the work does not cover diffusion-based or hybrid generative models, more recent large-scale systems, more extreme regimes, or larger sample budgets. Additionally, unimodality of temperature profiles is empirically supported and given a structural justification, but not proven universally — and the Short-Sampling Effect has no single unified theorem covering all settings.

Target Audience

Researchers working on inference-time compute, test-time scaling, and LLM sampling methodology; practitioners who design multi-sample serving pipelines for code generation or mathematical reasoning and need to justify a sampling configuration; and theoretically inclined readers interested in optimization under resource constraints. The paper is most valuable to those who have encountered claims that multi-temperature sampling yields large gains and want a rigorous account of when that is and is not true. Readers without a background in convex optimization or asymptotic analysis can still follow the empirical findings and the practical recommendations from Sections 3 and 4.3, but the proofs in Appendix B require comfort with the underlying mathematics.

Authors’ abstract

Inference-time sampling is a central mechanism for improving large language models, with temperature controlling the tradeoff between reliability and diversity. While recent work suggests that varying temperature across samples can improve multi-sample inference, simple fixed-temperature schemes remain the standard in practice. We investigate this discrepancy in the canonical pass@K setting, asking when temperature diversity is useful, and when a single tuned temperature…