Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions

AI summary

Overview

• Research area: Natural language processing / evaluation of large language models, specifically the statistical fidelity of LLM-generated random samples and the downstream consequences of sampling failure.
• Technical level: Advanced. The paper combines a large-scale empirical benchmark with distributional-statistics machinery (Wasserstein-1 distance, Kolmogorov–Smirnov and chi-square tests, KL divergence) and a formal decomposition of batch-generation error into correction and drift terms.
• Scope: The first large-scale, statistically powered audit of native probabilistic sampling in frontier LLMs, benchmarking 11 models across 15 distributions under two sampling protocols (N=1000 each), plus two downstream application tasks (MCQ generation and attribute-constrained text-to-image prompt synthesis).

What This Paper Is About

LLMs are increasingly used as components of pipelines that need random samples — generating synthetic data, building test questions, or writing image prompts to demographic targets. Currently, practitioners work around this by having models emit Python code that calls numpy.random, which implies the models themselves never learned to sample. This paper asks whether current frontier LLMs can natively produce samples matching a specified probability distribution, at a sample size large enough to make statistical conclusions trustworthy.

Key Contributions

1. The authors demonstrate that current LLMs lack a functional internal mechanism for probabilistic sampling, showing a sharp protocol asymmetry: batch generation reaches a 7% median pass rate, while under independent requests 10 of the 11 models pass none of the distributions.
2. They reveal that sampling performance is bounded by distributional complexity, with pass rates declining and Wasserstein-1 distances rising across three distribution tiers.
3. They identify inverse scaling: increasing the sampling budget N degrades distributional adherence rather than improving it, contradicting the standard O(N^(-1/2)) convergence rate expected of an ideal i.i.d. sampler.
4. They provide the first systematic evidence that native sampling failures propagate into downstream tasks, showing severe answer-position bias in LLM-generated multiple-choice questions and systematic violation of demographic and attribute targets in text-to-image prompt synthesis.

Main Findings

• Protocol asymmetry is severe. Batch generation (N=1000 samples in one response) achieves only modest statistical validity, with a 7% median pass rate and the leading model passing 40% of distributions. Under independent requests (N=1000 stateless calls), 10 of the 11 models pass none of the distributions; only Llama-4 reaches a 7% pass rate.
• The asymmetry is not explained by task difficulty. On Uniform, the simplest benchmark, Wasserstein distances amplify from W1 ≈ 0.01 in batch mode to W1 ≈ 0.15 in independent mode across models. The authors argue valid sampling depends on long-context dependency rather than on an intrinsic sampling ability.
• Fidelity degrades with distributional complexity. Across the three tiers (Tier I: Uniform, Gaussian, Bernoulli; Tier II: Beta, Binomial, Poisson, Exponential; Tier III: Cauchy, Student's t, Chi-Square, F, Gamma, Weibull, Laplace, Logistic), pass rates decline monotonically while mean W1 rises from ~0.1 (Tier I) to ~1.5 (Tier III). No model passes the statistical tests on heavy-tailed distributions such as Cauchy and Chi-Square. GPT-4o shows the steepest degradation from perfect Tier I performance.
• Inverse scaling with sampling budget. Sweeping N ∈ {50, 100, 200, 300, …, 1000, 1500, 2000} for Gaussian and Bernoulli with DeepSeek-V3.2 and Kimi-K2, batch generation shows a regime shift: after early improvement at short horizons, W1 turns upward and increases with N, with KS p-values collapsing below the α=0.01 threshold. Independent requests fail the KS test at all N yet still drift upward in W1.
• Batch-mode success is contextual scaffolding, not competence. The authors attribute batch-mode pass rates to within-context self-correction (their "Correction Gain" term) rather than to an internalized understanding of distributions, with drift (exposure bias) eventually outweighing correction — the "Context–Fidelity Dilemma."
• MCQ generation shows severe positional bias despite explicit instructions. With N=1000 medical questions generated via independent calls and prompts instructing that correct answers be uniform across A/B/C/D (25% each), all six tested models show statistically significant bias (p < .001). GPT-OSS-120B places 54.6% of correct answers at position C and only 4.5% at A (chi-square 577.2). GPT-4o favors B at 46.8% and nearly ignores D at 5.5% (chi-square 444.5). No model approaches 25% for any position.
• Attribute-constrained prompt generation systematically violates targets. Across N=1000 text-to-image prompts per model, GPT-4o overrepresents males (75.0% vs. target 49.5%) while Llama-4 overrepresents females (97.2%). For race/ethnicity, GPT-4o produces 33.5% Asian (target 6.5%) and 0% Hispanic (target 20.0%); the "Other" category is 0% for four of six models against a 3.4% target. Height variance collapses to σ ≈ 1–6 cm against a target σ = 10 cm, with KS statistics exceeding 0.37 for all models. For coat color, Llama-3.3 generates 96% green coats and GPT-OSS favors red at 54%, against a uniform 14.3% per category.
• Decoding sweeps do not rescue native sampling. Appendix ablations over temperature ∈ {0.2, 0.5, 1.0, 1.2} and top-p ∈ {0.9, 0.95, 1.0} for GPT-4o and Gemma-3 across all 15 distributions leave the overall conclusion unchanged, and the independent protocol remains broadly poor.

Methodology in Plain English

The authors treat the LLM as a candidate sampler. For each of 15 target distributions, they ask a model for N=1000 numbers and then compare the resulting sample set to a high-precision reference sample of N=1000 drawn from numpy.random and scipy.stats.

Two prompts are used to separate failure modes. In the batch protocol, one prompt asks for all 1000 samples in a single response, so each number is written after the model has already seen its own earlier outputs. In the independent protocol, the same model is called 1000 separate times, each time asking for exactly one number with no shared context, so no history can influence a sample.

Comparison uses three families of measurement: the Wasserstein-1 distance (the L1 distance between the empirical and target CDFs), a histogram-based KL divergence, and a significance test with α = 0.01 — a two-sample Kolmogorov–Smirnov test for continuous distributions and a chi-square goodness-of-fit test against the theoretical PMF for discrete ones. The strict threshold is chosen because N=1000 gives high statistical power.

To trace how error accumulates, they sweep sample sizes from 50 up to 2000 for Gaussian and Bernoulli with DeepSeek-V3.2 and Kimi-K2 under both protocols, recording W1, KL, and the relevant test statistic at each checkpoint.

They then test whether these abstract failures matter in practice, running two downstream tasks with six representative models: generating 1000 medical multiple-choice questions with explicit instructions to distribute correct answers uniformly across A/B/C/D, and generating 1000 photo prompts describing a person in a coat, with prescribed gender and race/ethnicity targets drawn from U.S. Census Bureau (2024) figures, a height target of N(169, 10²) cm, and a coat color uniform over seven categories.

Finally, they formalize the two regimes. Under independent requests, the expected error stabilizes at Δ_ind + O(N^(-1/2)), an irreducible bias floor. Under batch generation, they decompose error into a Correction Gain and a Drift (exposure bias) term, showing why net fidelity can worsen as context grows.

Why This Matters

Impact on research. The paper reframes distributional sampling as a functional competence test rather than a curiosity. Its sample-size scaling result — that larger N exposes rather than resolves the mismatch — implies that small-N studies such as the earlier N=100 evaluation can report misleadingly optimistic conclusions, and that future benchmarks need sufficient power to detect these failures. The dual-protocol design also offers a template for distinguishing an internalized capability from an artifact of autoregressive conditioning.

Real-world applications:

• Educational assessment. LLM-generated multiple-choice tests can concentrate correct answers in specific positions, letting test-takers exploit positional patterns — a direct threat to test validity in automated question generation.
• Synthetic data and text-to-image datasets. Prompts that are supposed to hit demographic targets instead collapse onto modal preferences, embedding sampling-induced bias directly into training data intended for fairness purposes.
• Agent-based simulation and world models. Simulations that assume behavioral diversity from LLM sampling inherit systematic internal biases that are not grounded in the intended distribution.
• Randomized decision-making pipelines. Any downstream system treating LLM outputs as valid probabilistic samples has an unwarranted assumption at its base.

Industry relevance. The paper's practical recommendation is that applications requiring statistical guarantees need external sampling tools (numpy.random, scipy.stats) rather than native model sampling. Teams that currently generate random numbers, balanced test sets, or demographic prompt sets directly from a model should treat their outputs as biased unless validated by a statistical test.

Future Directions

• Determine whether the failure is intrinsic to current training paradigms or can be addressed by architectural or training changes; the authors explicitly note their results are empirical rather than an impossibility proof for future architectures.
• Extend the benchmark beyond explicitly specified 1D distributions to implicit or context-dependent stochastic processes, which the limitations section identifies as outside the current experimental scope.
• Investigate whether interventions that improve within-context self-correction can delay or eliminate the drift that causes inverse scaling as N grows.
• Conduct comprehensive fairness analyses of downstream applications; the paper presents its MCQ and text-to-image tasks as controlled tests of sampling-induced bias rather than as a full treatment of downstream fairness.

Target Audience

Researchers and practitioners working on LLM evaluation, synthetic data generation, educational content automation, text-to-image dataset construction, and agent-based simulation. It is most useful to readers with working familiarity with probability distributions and hypothesis testing, since the core evidence is presented as Wasserstein distances and p-values. Engineers building production pipelines that depend on model-generated randomness will find the practical prescription — use external samplers for statistical guarantees — immediately actionable.

Authors’ abstract

As large language models (LLMs) transition from chat interfaces to integral components of stochastic pipelines and systems approaching general intelligence, the ability to faithfully sample from specified probability distributions has become a functional requirement rather than a theoretical curiosity. We present the first large-scale, statistically powered audit of native probabilistic sampling in frontier LLMs, benchmarking 11 models across 15 distributions. To disentangle failure modes, we employ a dual-protocol design: Batch Generation, where a model produces $N{=}1000$ samples within one response, and Independent Requests, comprising $N{=}1000$ stateless calls. We observe a sharp protocol asymmetry: batch generation achieves only modest statistical validity, with a 7% median pass rate, while independent requests collapse almost entirely, with 10 of 11 models passing none of the distributions. Beyond this asymmetry, we reveal that sampling fidelity degrades monotonically with distributional complexity and aggravates as the sampling horizon $N$ increases. Finally, we demonstrate how the propagation of these failures into downstream real-world application tasks introduces systematic biases: models fail to enforce uniform answer-position constraints in Multiple Choice Question generation and systematically violate demographic targets in attribute-constrained text-to-image prompt synthesis. These findings indicate that current LLMs lack a functional internal sampler, necessitating external tools for applications requiring statistical guarantees.