Certain but not Probable? Differentiating Certainty from Probability in LLM Token Outputs for Probabilistic Scenarios

AI summary

Overview

Research area: Natural Language Processing / Large Language Model uncertainty quantification (UQ), with a focus on probabilistic reasoning and calibration.

Technical level: Intermediate. The paper assumes familiarity with token logits, softmax-derived probabilities, and entropy, but defines all key terms and reports its comparisons in a table-driven, intuitive way.

Scope: Ten well-defined probabilistic prompts (die rolls, coin flips, card draws, and similar scenarios) are posed to two closed-source LLMs to test whether token-level probability and entropy values align with the theoretical distributions those scenarios imply.

What This Paper Is About

When an LLM answers a prompt like "roll a six-sided die," users want to know how confident the model is. The usual way to measure that confidence is to read the probability the model assigned to the token it emitted, and to compute entropy over its token distribution. But in a genuinely random scenario, a trustworthy model's output distribution should actually match the real-world distribution — a fair die should come up each value with probability 0.167, yielding an entropy of 2.58. The authors ask whether the model's token-level certainty behaves that way, or whether the model can be perfectly well-behaved in terms of validity (never saying "7" on a six-sided die) while being badly miscalibrated in terms of probability.

Key Contributions

1. A separation of two evaluation dimensions for probabilistic prompts. The paper argues that uncertainty quantification for probabilistic scenarios requires distinguishing (a) whether a response is valid under the scenario's constraints from (b) whether the response probability aligns with the underlying theoretical distribution. Traditional UQ conflates low uncertainty with correctness; here, a model can be certain and correct while still miscalibrated.

2. A controlled prompt-design framework covering specified and unspecified scenarios. Ten scenarios are used, spanning chance actions (coin flip, die roll, roulette, rock-paper-scissors, bingo) and random selection from sets of items (Bible book, month and day, playing cards, Shakespeare play, dart on a map). Each is run in an unspecified form and a specified form that adds the distributional cue — "fair" for coin flip, die roll, and rock-paper-scissors; "uniformly at random" for Bible, bingo, month and day, playing cards, rock-paper-scissors, and Shakespeare; "randomly" for dart on a map and roulette. The prompt "Respond only with the result" is appended to all prompts so that generated output does not itself perturb the token probabilities, and the probabilistic constraints are not spelled out in the base versions, avoiding the prompt-induced bias noted in prior work.

3. Empirical comparison of GPT-4.1 and DeepSeek-Chat against theoretical distributions. Using gpt-4.1-2025-04-14 and DeepSeek-R1-0528 with logprobs=true (returning the top 20 tokens and their logprob values), five independent samples per prompt per model are collected, and mean token probability and entropy are compared to the analytic values for each scenario.

4. Three framing research questions for extending UQ to probabilistic tasks, covering whether LLMs can reason about theoretical probabilities, whether they are reliable stand-alone tools for probability-oriented tasks, and how UQ metrics could jointly evaluate validity and distribution alignment.

Main Findings

• Perfect validity, imperfect calibration. Both GPT-4.1 and DeepSeek-Chat produced responses within scenario constraints across all samples, at 100% response accuracy. The only caveat noted is DeepSeek-Chat returning partial responses in Bingo and Roulette (a bare number such as 17 or 14, rather than GPT-4.1's letter-plus-number or number-plus-color such as G-52 or 27 Black); the authors treat these as accurate since the numeric values are valid options.

• Unspecified prompts are systematically overconfident. For every unspecified prompt, both models produced token probability values higher and entropy values lower than the corresponding theoretical values. For the "heads" token in the coin flip, both models returned Pr(t) = 1.0 with H(T) ≤ 0.0002, against a theoretical 0.5 and 1.0. DeepSeek-Chat produced 0.979 probability and 0.144 entropy for "Hamlet" in the Shakespeare scenario (theoretical 0.025 and 5.29).

• Specification helps only a little and inconsistently. Adding the distributional cue moved values in the expected direction — decreasing probability and increasing entropy — but the rate of correction was inconsistent and the specified values remained far from theoretical. The largest shift was in the Shakespeare scenario: DeepSeek-Chat fell from 0.979 to 0.4 in probability and rose from 0.144 to 1.1 in entropy; GPT-4.1 fell from 0.66 to 0.20 and rose from 0.89 to 2.98.

• Not all corrections point the same way. GPT-4.1 generally showed lower probability and higher entropy under specification, though roulette moved the other way (0.13 to 0.16 probability, 3.39 to 3.34 entropy), and the dart-on-map (3.47 to 3.00) and rock-paper-scissors (1.29 to 0.66) entropies actually fell. DeepSeek-Chat showed increased probability in four scenarios: die roll (0.3 to 0.593), playing cards (0.6 to 0.994), rock-paper-scissors (0.7 to 0.978), and roulette (0.08 to 0.2); three of those four also dropped in entropy (die roll 1.24 to 1.160, playing cards 0.289 to 0.061, rock-paper-scissors 0.5 to 0.15).

• Against theory, GPT-4.1 aligns better but still fails. Using the specified prompts as the comparison basis, GPT-4.1's average probability differences exceed -0.2 in seven of ten scenarios, while all of DeepSeek-Chat's probability differences fall below that threshold. GPT-4.1's best alignments were Bingo (-0.03) and Month-and-Day (-0.02); DeepSeek-Chat's best was roulette (-0.22).

• The simplest tasks are among the worst cases. GPT-4.1's poorest probability alignment came on the die roll (-0.75) and coin flip (-0.50). DeepSeek-Chat's worst was playing cards (-0.97), where it always responded "7 of hearts" under the specified prompt.

• Entropy is poorly captured across the board. The entropy associated with a coin flip deviates by nearly 100%, and both models show higher than 30% error in modeling entropy in all scenarios.

• Repeated, deterministic answers. Across the five samples per configuration, certain answers recurred heavily: both models answered "heads" for the coin flip and "4" for the die roll in all ten combined samples, and both predominantly chose "scissors" in rock-paper-scissors (9 occurrences for DeepSeek-Chat, 6 for GPT-4.1). Most other scenarios also had a modal response, with exceptions being GPT-4.1's Bingo and month-and-day samples.

• Verbalized reasoning does not match sampling behavior. In follow-up interactive chat, both models articulated the correct probability distributions for the scenarios, yet their response tokens carried probability and entropy values that did not align with those distributions — a gap between reasoning explanations and probabilistic calibration.

Methodology in Plain English

The authors picked ten familiar scenarios where the correct probability distribution is unambiguous and can be written down in advance (Table 1 lists each scenario's theoretical outcome probability and entropy under a uniform distribution). For each scenario they wrote two prompts: a plain version ("Flip a coin.") and a version that hints at the distribution ("Flip a fair coin."). Both GPT-4.1 and DeepSeek-Chat were queried through the Chat Completions API with logprobs=true, requesting the top 20 tokens and their logprob values. Because vendors expose only the top 20 tokens, the authors add an "other" token with a set probability so the distribution sums to 1; they note that this, plus the fact that logprobs are given only for the selected token sequence rather than all possible sequences, likely makes the computed entropy a slight underestimate of the model's true entropy. Token probability is recovered as e^(l(t)) and entropy is computed as H(T) = −Σ log₂(Pr(t)) · Pr(t) over the token set. Five independent samples are drawn per prompt per model, mean probability and entropy are averaged, and those averages are compared first between the specified and unspecified prompt versions, then against the theoretical values from Table 1.

Why This Matters

The paper's central point is that a model can look accurate and feel certain while being systematically miscalibrated, and that ordinary UQ metrics will not reveal this. "Apparent accuracy may mask deficiencies in its probabilistic calibration." Anyone building on LLM outputs in domains with real randomness needs an evaluation dimension that ordinary confidence scores do not provide.

Real-world applications the authors identify:

• Simulations that rely on LLM-generated random draws.
• Randomized trial designs, where misaligned sampling introduces systematic bias.
• Decision-support systems involving chance, where the authors warn that miscalibration "could lead to systematic bias or misleading outcomes."
• Larger processing pipelines in which LLMs act as components rather than stand-alone tools, where the paper argues UQ is important not only for confidence estimates but for understanding what those estimates represent.

Industry relevance: Both models tested are closed-source frontier systems with explicit logprob support, so the failure mode is not an artifact of an obscure setup. The authors conclude that LLMs in their current form may not be suitable as stand-alone tools for probability-oriented tasks without "additional calibration, fine-tuning, or post-processing to enforce distributional alignment," and that integrating them into probabilistic workflows should require evaluation beyond traditional UQ before deployment. The results echo prior work by Lovering et al. (2025) despite a different prompt style, and the authors note that by stripping responses down to the final answer alone, they show the bias persists even without sequential generation effects.

Future Directions

1. Can LLMs accurately reason about theoretical probabilities? The follow-up chats suggest the underlying knowledge is present, since both models articulated correct distributions, yet the emitted tokens did not reflect it. How to close that reasoning-to-sampling gap remains open.

2. Are LLMs appropriate and reliable for probability-oriented tasks at all? If usability depends on matching a theoretical distribution, the paper's evidence points to no without intervention — leaving open what form that intervention (calibration, fine-tuning, post-processing) should take.

3. How should UQ be adapted? The authors recommend either separate metrics — one for validity, one for probability alignment — or a single composite score combining both, so that UQ can express the "certainty" of distribution alignment rather than only the certainty of a token.

4. Extending beyond these ten scenarios. The work uses ten well-known scenarios with analytic distributions; whether the same divergence holds across broader probabilistic tasks, additional models, and multi-token outputs is not resolved here.

Target Audience

Researchers and practitioners working on LLM uncertainty quantification, calibration, and evaluation; engineers integrating LLMs into simulations, randomized designs, or decision-support systems where probabilistic behavior matters; and anyone designing confidence or reliability signals for model outputs. The paper is readable by readers with a basic grasp of token probabilities and entropy, and its code is released at https://github.com/autumntoney/chatbot-certainty.

Authors’ abstract

Reliable uncertainty quantification (UQ) is essential for ensuring trustworthy downstream use of large language models, especially when they are deployed in decision-support and other knowledge-intensive applications. Model certainty can be estimated from token logits, with derived probability and entropy values offering insight into performance on the prompt task. However, this approach may be inadequate for probabilistic scenarios, where the probabilities of token outputs are expected to align with the theoretical probabilities of the possible outcomes. We investigate the relationship between token certainty and alignment with theoretical probability distributions in well-defined probabilistic scenarios. Using GPT-4.1 and DeepSeek-Chat, we evaluate model responses to ten prompts involving probability (e.g., roll a six-sided die), both with and without explicit probability cues in the prompt (e.g., roll a fair six-sided die). We measure two dimensions: (1) response validity with respect to scenario constraints, and (2) alignment between token-level output probabilities and theoretical probabilities. Our results indicate that, while both models achieve perfect in-domain response accuracy across all prompt scenarios, their token-level probability and entropy values consistently diverge from the corresponding theoretical distributions.