Rare Event Analysis of Large Language Models

AI summary

Overview

Research area: Machine learning / large language model safety and evaluation, drawing methods from statistical physics and computational chemistry (rare event sampling, MCMC, free-energy estimation).

Technical level: Advanced. The paper is written as a practical guide, but it assumes familiarity with importance sampling, Metropolis-Hastings, partition functions and confidence-interval construction.

Scope: The paper presents an end-to-end framework — setup, probability estimation and exploration — for analysing rare events in large language models, demonstrated on the TinyStories-8M model with two observables.

What This Paper Is About

LLMs are probabilistic models, so during inference they sometimes produce outputs that are highly atypical but important — rare events. Because these events are by definition almost invisible during testing, yet LLM deployment happens at enormous scale, outputs that were never observed in development can become prominent in real use. The paper's goal is to provide a practical, methodical framework for defining, estimating the probability of, and exploring such rare completions, rather than relying on the current default of direct sampling, which almost never sees the tails of the distribution.

Key Contributions

1. The authors present what they describe as the first complete end-to-end application of rare event analysis (REA) to LLMs, organised into three stages: setup, estimation and exploration.
2. They provide a practical implementation guide covering theory, efficient generation strategies, probability estimation and error analysis.
3. They analyse the probabilities of rare completions in TinyStories models for two observables of practical interest: the automated readability index (ARI) and the log-probability of a completion (Log-Prob).
4. They give an example of exploratory data analysis (EDA) of actual rare completions, and outline directions for improving the algorithms and applying them to other models and contexts.
5. A minimal code implementation of the framework is referenced for reproducibility (McAllister Dorman et al., 2026).

Main Findings

• Direct sampling cannot reach the tails: Direct sampling is described as inefficient because it generates far more samples than needed for typical values and too few (or no) samples for atypical values. It is identified as the current "state-of-the-art" for rare completion probability estimation, and the only directly comparable method to the setup used here.
• MBAR reconstruction accesses far deeper tails: Using MBAR to combine TPS samples, the resulting histograms contain counts across the range and probe densities orders of magnitude smaller than direct sampling. For ARI, the direct histogram is limited to the range of the training data, while the MBAR histogram extends well beyond it.
• The distributions are strongly non-Gaussian: The paper notes this as a common trait when correlations are present, as here due to sequential token generation, leading to increased likelihood of rare outcomes.
• Smaller relative error bars in the tails: Using 96% confidence intervals, the MBAR estimates show significantly smaller relative CI widths than direct sampling in the tails. The authors caution that rigorous interpretation in terms of coverage probabilities requires care.
• Bias analysis suggests more steps per chain is the better investment: Comparing the change in bin-height estimates when doubling the number of MCMC steps, the ratio is generally small but some values are close to 1 in the tails, indicating tail results may change substantially with more steps while likely remaining of the right order of magnitude. The change varies from comparable to slightly smaller than the CI half-widths, suggesting that for this setup increasing steps within each MCMC chain could reduce statistical error more than increasing the number of independent chains.
• A concrete rare-event example: High-ARI, high-probability completions were found beyond the training-data regime; one reads "eros. He had a friend named Trurururu. Trururururururu was very slow, so the other rhinrururururururururururu complained to Trurururururu… ['ru' continues 50 more times]". Despite being out of distribution relative to the training data and unlike human text, it is assigned high probability by the model.
• Repetition as a cheap proxy: Highly repetitive text appears in extreme-ARI completions, motivating consecutive token repeats as a candidate low-cost proxy observable for early filtering. The bulk of the distribution shows only a weak negative correlation between ARI and Log-Probability, consistent with the training data.
• Interpretation of the extrapolation behaviour: The authors suggest that when extrapolating outside the regime of the training data, the model falls back on basic patterns favouring repetition to yield acceptable likelihoods.

Methodology in Plain English

The paper treats a completion as a trajectory: the prompt is fixed, and the model generates tokens one at a time, so a full completion has a well-defined probability. Two functions of the prompt–completion pair are then defined as "observables": the Log-Prob (how likely the completion is under the model) and the ARI (a readability score). Rare events are completions where an observable takes an extreme value.

Because direct sampling almost never produces such completions, the authors bias generation towards them. They use exponentially reweighted ("tilted") distributions, where a parameter lambda controls how strongly the sampling is pushed towards extreme observable values, and they run positive and negative biases to target both tails. Samples are drawn with Transition Path Sampling (TPS), a Metropolis-Hastings variant suited to sequences: a trajectory is truncated at some point and regenerated autoregressively, with proposals accepted or rejected. The bias is increased gradually along an annealing schedule to help the chains converge.

To turn biased samples back into honest probabilities, the authors use importance sampling with a mixture of biased distributions — umbrella sampling — and solve for the unknown normalisation constants (partition functions) using MBAR, implemented through the pymbar package. Error bars come from percentile bootstrap over independent chains, and the Gelman-Rubin statistic is used to discard non-converged samples. For the exploration stage, the authors apply EDA-style analysis to the actual rare texts that were generated.

Specific setup: TinyStories-8M, ancestral ("temperature 1.0") completion of 100 tokens from the fixed prompt "Once upon a time, in a big forest, there lived a rhinoc", which is the first 16 tokens of an entry in the TinyStories validation split. The ARI is computed over the whole prompt–completion pair with a cap at 15, motivated by ARI's common use as an integer between 1 and 14 and to improve MCMC acceptance rates. The annealing schedule uses 10 bias values increasing in magnitude, each run for 4×10⁴ TPS steps, totalling 4×10⁵ samples per bias; the first 10% of each chain is discarded as burn-in, and samples from any bias with a Gelman-Rubin statistic greater than 1.1 are discarded. On average 50 tokens are generated per TPS step, giving approximately 4×10⁸ tokens generated. Direct sampling produced 4.2 million completions, roughly 420 million tokens per observable. MBAR estimates combined the TPS samples with an additional 2×10⁵ directly sampled completions, totalling about 7×10⁶ out of about 8×10⁶ generated completions, again exceeding 4×10⁸ tokens per observable to allow a fair comparison. The reported histograms required of the order of 10⁹ tokens; the authors note that Google reports Gemini processing of the order of 10¹⁵ tokens per month, so equivalent histograms for Gemini would require about one second of its total worldwide processing. Confidence intervals are 96%, built from 100 bootstrap replicas, with the smallest and largest third percentile of the estimates giving the bounds; Wilson intervals are used for direct sampling. For the exploratory analysis, an additional 2.1×10⁶ completions were generated with the same annealing schedule but a reduced 2×10⁴ steps per bias.

Why This Matters

Impact on research. The paper argues that rare event analysis for LLMs is still in its infancy, and that methods from statistical physics and computational chemistry — umbrella sampling, MBAR, TPS, and related Monte Carlo techniques — transfer usefully to language models. It draws explicit links to alignment objectives such as RLHF and DPO, which share the structure of a time-integrated reward with a KL-divergence regularisation, and notes that a recent "distribution sharpening" approach is directly analogous to TPS's shooting method, while another estimates a modified "Doob transform" distribution. The paper positions itself relative to Wu and Hilton (2025), which estimates the probability of a single rare token by varying the prompt, and Jones et al. (2025), which extrapolates rare event probability estimates from a small set of test prompts to a much larger set of deployment prompts for Claude 3/3.5. The authors frame their accurate few-prompt estimates as complementary to that extrapolation work.

Real-world applications.

• Safety guardrails and filters for deployed models, where cheap proxies (such as token repetition) can be monitored at runtime to flag rare, undesirable completions before a full generation is evaluated.
• Regulatory and compliance contexts: the paper notes that related readability measures are used in regulatory guidance, citing Chapter 8 of the FCA (2022) reference, making extreme readability scores of practical interest.
• Model evaluation and testing, giving developers a way to quantify how often a model fails even when those failures are essentially unobservable in ordinary testing.
• Decoding algorithm development, since extreme values of the log-probability and repetition behaviour are directly relevant to how text is generated.

Industry relevance. The authors state the framework is intended for developers, researchers and engineers working on large foundation models as well as application builders in critical domains. They estimate the computational cost of accurate tail histograms — around 10⁹ tokens — and note that for a flagship-scale model such as Gemini, whose reported processing volume is about 10¹⁵ tokens per month, the equivalent would take roughly one second of total worldwide processing.

Future Directions

• Scaling beyond a small model: The authors explicitly state that the ultimate implementation for industry-standard LLMs will require widespread collaboration of experts and deep technical contributions across fields. TinyStories-8M was chosen as a midpoint between a fully solvable toy problem and a realistic large-scale LLM.
• Reducing statistical bias and variance: The paper suggests that, for this setup, increasing the number of steps within each MCMC chain may reduce overall error more than adding more parallel chains, and calls for improvement of the algorithms more generally.
• Domain-specific exploration: Rare event exploration is described as highly domain dependent — for example, generating dangerous completions to aid development of specialised guardrails and filters — so the generic EDA demonstrated here is presented as a starting point for problem-specific investigations.
• Better proxies and observables: The paper raises the question of choosing a biasing observable that is a cheaper proxy correlated with an expensive target observable, an approach it says may be preferable when the target observable is costly to compute.
• Applications to other models and contexts: The authors outline extensions and applications to other models and contexts, highlighting the generality of the concepts and techniques. Specific open questions the framework poses include how likelihood, properties and efficient access to rare events generalise beyond the two observables studied, and how few-prompt estimates can best be combined with extrapolation to deployment-scale prompt sets.

Target Audience

ML engineers and researchers working on LLM safety, evaluation and decoding; application builders in critical domains who need to bound the frequency of rare failures; and statistical physicists or computational chemists interested in transferring rare event methods to language models. Some background in Monte Carlo methods and statistical estimation is needed to follow the derivation and error analysis, though the concrete TinyStories example and the referenced code implementation lower the barrier for practitioners who want to reproduce the workflow.

Authors’ abstract

Being probabilistic models, during inference large language models (LLMs) display rare events: behaviour that is far from typical but highly significant. By definition all rare events are hard to see, but the enormous scale of LLM usage means that events completely unobserved during development are likely to become prominent in deployment. Here we present an end-to-end framework for the systematic analysis of rare events in LLMs. We provide a practical implementation spanning theory, efficient generation strategies, probability estimation and error analysis, which we illustrate with concrete examples. We outline extensions and applications to other models and contexts, highlighting the generality of the concepts and techniques presented here.