On the Entropy Calibration of Language Models

AI summary

Overview

Research area: Natural Language Processing — calibration and generation quality of autoregressive language models, with a theoretical component drawn from learning theory and reinforcement learning.

Technical level: Advanced. The paper combines a stylized probabilistic analysis (singleton mass under power law distributions), empirical scaling-law measurements across model families, and a polynomial-time algorithm with a theorem on entropy calibration error.

Scope: The paper asks whether entropy miscalibration in language models shrinks automatically with scale, and whether it is theoretically possible to calibrate a model's entropy without paying a diversity cost.

What This Paper Is About

A language model is "entropy calibrated" if the uncertainty (entropy) of its own generations matches the log loss it incurs on real human text. Prior work (Braverman et al., 2020) found that this fails: as a model generates more tokens, its per-step entropy drifts upward relative to log loss, because errors feed back into the context autoregressively. The paper asks two questions: does this miscalibration improve automatically as models get bigger, and if not, can it be fixed without the usual fix of truncating the distribution, which reduces output diversity?

Key Contributions

1. A theoretical scaling prediction. The authors study a simplified setting where instability arises from generating a token seen only once during training, and derive that the per-step probability of generating such a singleton scales as $m^{1/\alpha - 1}$ with $m$ training examples and power law exponent $\alpha$ (Proposition 3.1). For $\alpha$ near 1, as is typical for text, this exponent is near 0.

2. Empirical scaling measurements from 0.5B to 72B parameters. Four model families (Qwen2.5, Llama 3, Llama 2, Pythia) across three datasets (WikiText-103, WritingPrompts, CodeContests), measuring how calibration error changes with model size.

3. A characterization of the calibration-diversity tradeoff. Temperature reduction and instruction tuning are both shown to reduce entropy while increasing log loss — calibrating at the cost of diversity.

4. A theoretical calibration procedure with a guarantee. Under an assumption about fitting regression models that generalize, the authors prove that a polynomial-time "future entropy scaling" procedure can calibrate the model while keeping log loss no worse than the original model (Theorem 5.2). The procedure is described as impractical to implement.

Main Findings

• Miscalibration barely improves with scale on text. The fitted scaling exponents for the two text datasets are reported in the abstract as around -0.05. Figure 1 lists per-dataset predicted exponents of 0.089 (WikiText), -0.10 (WritingPrompts), and -0.33 (CodeContests).

• Observed exponents depend on model family. For the older families (Llama 2 and Pythia), exponents are around 0.0 for WikiText and WritingPrompts and -0.2 for CodeContests. For the newer families (Llama 3 and Qwen2.5), they are around -0.13 for WikiText and WritingPrompts and -0.35 for CodeContests.

• The data heavy-tailedness predicts the scaling. Measured power law exponents are 0.918 for WikiText, 1.114 for WritingPrompts, and 1.5 for CodeContests. The code dataset's more quickly decaying tail is consistent with its faster scaling.

• Entropy per step rises with generation length, unlike human text. Log loss on ground-truth text is mostly constant or slightly decreasing over time, replicating the entropy rate constancy principle (Genzel and Charniak, 2002; Verma et al., 2023), while model generation entropy increases with the number of steps. Larger models grow at similar rates for WikiText and WritingPrompts; the slopes decrease with model size for CodeContests.

• Practical consequence of slow scaling. The paper states that a scaling exponent of -0.10 means reducing calibration error by a factor of 10 would require increasing dataset size by a factor of 10^10.

• Both truncation and instruction tuning trade diversity for calibration. Temperatures of 1.0, 0.95, 0.9, 0.85, and 0.8 reduce entropy but increase log loss, and the temperature attaining zero calibration error is similar across model sizes. Instruction tuning also reduces entropy and raises log loss, with larger models showing larger effects in the Qwen2.5 comparison, though the paper notes this pattern is not robust across model families.

• Calibration without log loss increase is provable in principle. Theorem 5.2 states that the entropy calibration error of the adjusted model is at most $2T\delta + \sum_{t=1}^{T}(1+\alpha_t)\varepsilon$ while its log loss is at most that of the original model, where $\delta$ is the future entropy predictor's test error and each $\alpha_t$ is an $\varepsilon$-stationary point.

• Future entropy adjustment is derived as a first-order approximation of globally normalized temperature scaling, which Braverman et al. (2020) showed can calibrate while preserving log loss but is intractable to compute over an exponentially large output space.

Methodology in Plain English

Theory first. The authors build a toy model of error accumulation: a model stores counts of $m$ tokens drawn from an $\alpha$-power-law distribution over a vocabulary of size $v$. If everything in context was seen at least twice, the model samples a familiar token; if any context token was seen only once, the model samples from a high-entropy "derailed" distribution. Because the rare-token probability scales as $m^{1/\alpha - 1}$, the expected entropy per step grows at that rate. They verify the asymptotic expression against finite simulations, finding the simulated slopes are close as long as $m$ is smaller than $v/3$.

Measurement second. The authors run four model families — Qwen2.5 (0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B), Llama 3 (1B, 3B, 8B, 70B), Llama 2 (7B, 13B, 70B), and Pythia (410M, 1.4B, 2.8B, 6.9B, 12B) — on three datasets: WikiText-103, WritingPrompts, and CodeContests. Each setting uses 5000 examples with samples limited to 1024 tokens and 128 tokens of context. Calibration error is the difference between average entropy per generation step and average log loss on ground truth. They plot log calibration error against log model size and fit a line, paralleling the scaling-law methodology of Kaplan et al. (2020) and Hoffmann et al. (2022). They also plot per-step entropy against per-step log loss, and separately compare temperature settings and base versus instruction-tuned Qwen2.5 models.

Theory of a fix third. The proposed adjustment reweights each candidate token's probability by both its log probability and a predicted "future entropy" — the entropy over everything the model would generate after that token. Because this quantity is intractable to compute exactly, the algorithm fits a predictor for it at each generation step and each candidate token, working backward from the last step to the first, and proves that the resulting model is calibrated up to the predictor's error while its log loss does not increase.

Why This Matters

Impact on research. The paper reframes error accumulation as a quantitative, measurable calibration property and connects it to the heavy-tailed statistics of text. It also gives a target — future entropy — for anyone designing decoding or sampling methods, arguing that current truncation heuristics are approximations to it. It identifies the calibration-diversity tradeoff as a concrete research gap rather than an inherent constraint.

Real-world applications:

• Test-time scaling and answer aggregation, where sampling many diverse generations and aggregating them is central (Wang et al., 2024; Brown et al., 2024).
• Synthetic data generation, which the paper links to community concern about running out of internet data (Wang et al., 2023; Gunasekar et al., 2023; Maini et al., 2024).
• Long-form generation such as story completion and code solution completion, the exact tasks used as the paper's three evaluation datasets.
• Understanding the alignment tax, since the paper's instruction-tuning results connect to prior findings that alignment can degrade capabilities (Ouyang et al., 2022; Bai et al., 2022; Lin et al., 2024).

Industry relevance. Practitioners choosing sampling temperatures and truncation thresholds now have evidence that the optimal truncation level does not change much with model size on text, and a reason why: larger models accumulate error at a similar rate. The paper also suggests that newer model families with different pretraining data mixtures — particularly a midtraining step with higher quality and less diverse data — scale somewhat better, though the authors note training details for three of the four families are not public.

Future Directions

1. Establish the power law / scaling exponent relationship more firmly across more datasets; the authors state that further work is needed to more strongly establish this relationship.

2. Disentangle model size, dataset size, and dataset composition using controlled data mixtures, since training details for Llama 3, Qwen2.5, and Llama 2 are not public.

3. Test how well large neural models actually predict future entropy, since the paper's guarantee assumes a fitted predictor with low test error but does not demonstrate this empirically. The authors note future entropy prediction involves mapping a set of tokens to a single bounded scalar, but the accuracy of a large neural model at this task is unknown.

4. Develop practical approximations of future entropy scaling, such as using one-step future entropy (Braverman et al., 2020) or truncating to $k$ steps, with the paper noting that worse approximations merely weaken the calibration guarantee.

Target Audience

Researchers working on language model generation quality, decoding and sampling methods, calibration, and scaling laws; theorists interested in learning-theoretic guarantees for autoregressive models; and practitioners who tune temperature, top-k, top-p, or min-p and want to understand what those knobs are actually doing to the entropy-diversity tradeoff. A reader needs comfort with probability, KL divergence, and scaling-law plots to follow the theoretical sections.

Authors’ abstract

We study the problem of entropy calibration, which asks whether a language model's entropy over generations matches its log loss on human text. Past work found that models are miscalibrated, with entropy per step increasing as generations grow longer, due to error accumulation. To calibrate the model and improve text quality, it has become standard practice to truncate the distribution, but this approach reduces output diversity, which we would like to avoid. Therefore, in this paper, we ask: does miscalibration improve automatically with scale, and if not, is it theoretically possible to calibrate without tradeoffs? To build intuition, we first study a simplified theoretical setting to characterize the scaling behavior of miscalibration with respect to dataset size. We find that the rate of scaling depends on the power law exponent of the data distribution -- in particular, for a power law exponent close to 1, the scaling exponent is close to 0, meaning that miscalibration improves very slowly with scale. Next, we measure miscalibration empirically in language models ranging from 0.5B to 70B parameters. We find that the observed scaling behavior is similar to what is predicted theoretically: our fitted scaling exponents for text are close to 0, meaning that larger models accumulate error at a similar rate as smaller ones. This scaling (or, lack thereof) provides one explanation for why we sample from larger models with similar amounts of truncation as smaller models, even though the larger models are of higher quality. However, truncation is not a satisfying solution because it comes at the cost of increased log loss. In theory, is it even possible to reduce entropy while preserving log loss? We prove that it is possible, if we assume access to a black box which can fit models to predict the future entropy of text.