Bayesian Evaluation of Large Language Model Behavior

AI summary

Overview

• Research area: Natural Language Processing / large language model evaluation, with a statistical (Bayesian) framing; published on arXiv as 2511.10661v1 [cs.CL] on 04 Nov 2025 by authors from the Department of Statistics and Department of Computer Science at the University of California, Irvine.
• Technical level: Intermediate (accessible to readers with basic probability and statistics background; the paper explicitly writes for "an applied statistics audience" and includes background primers on LLM text generation).
• One-sentence scope: The paper develops a Bayesian Beta-binomial model for binary LLM behavior metrics that accounts for uncertainty from stochastic text generation, and extends it to a sequential, bandit-style sampling scheme that chooses which prompts to sample next.

What This Paper Is About

Benchmark evaluations of LLM-based systems typically collapse many test prompts into a single score, often using deterministic (greedy) decoding or a single generation per prompt, and frequently neglect statistical uncertainty. This matters because LLMs are probabilistic systems: the same prompt induces a distribution over outputs, so a single observed output hides the underlying probability of a behavior such as refusing a harmful request or leaking information. The paper's goal is to place a Bayesian model over that per-prompt probability, propagate the resulting uncertainty into aggregate, policy-relevant metrics, and use the model to decide cost-effectively which prompts are worth sampling again.

Key Contributions

1. A Bayesian model for binary evaluation metrics under stochastic decoding. Each prompt's unknown probability of the behavior of interest is given an independent Beta prior, the repeated generations form a binomial likelihood, and conjugacy yields independent Beta posteriors — one per prompt.
2. Posterior distributions for aggregate behavioral metrics. The paper defines aggregation functions such as $W_{>\nu}=\sum_{m=1}^{M} I(\theta_m > \nu)$ (the number of prompts whose behavior probability exceeds a threshold) and $W_{\text{mean}}$ (the average probability across prompts), and derives or approximates their posterior distributions — closed form via a Poisson binomial for $W_{>\nu}$, Monte Carlo for $W_{\text{mean}}$.
3. Two applied case studies under a batch design. Pairwise preference between two LLMs on MT-Bench, and refusal behavior on JailBreakBench harmful prompts, each analyzed with equal numbers of generations per prompt.
4. A sequential (online) formulation of LLM evaluation. The evaluation problem is recast as a multi-armed bandit where arms are prompts, pulls are stochastic generations, and the reward is information gained about the aggregate $W$ (for example, reduction in $\mathrm{Var}(W)$), motivated by the financial and computational cost of repeated API calls. Thompson sampling is listed among the paper's keywords.

Main Findings

• Greedy decoding gives a single number; the Bayesian model adds a distribution. Under greedy decoding, Model A's response was preferred on 41/80 MT-Bench prompts. Under stochastic decoding with the Bayesian model, the 95% credible interval for $W_{\text{mean}}$ was (51%, 53%) — broadly in agreement with the greedy result, but expressed as a range.
• A single deterministic label cannot distinguish prompts with different underlying probabilities. The paper gives the example where $\theta_m = 0.6$ and $\theta_{m'} = 0.99$ under stochastic decoding, yet greedy decoding or a single generation shows Model A preferred in both cases, providing no information to differentiate the prompts.
• The threshold metric extracts information greedy decoding cannot. The mode of $W_{>\nu}$ under the Bayesian model estimates that with $n = 50$ generations, 23 prompts have Model A's response preferred with at least 75% probability (using $\nu = 0.75$ in the pairwise-preference case study).
• Refusal behavior is less uniform than greedy decoding suggests. On the M = 100 harmful prompts from the JailBreakBench Behaviors dataset, greedy decoding refused 98/100. The mode of $W_{>\nu}$ under the Bayesian model ($\nu = 0.95$, $n = 50$) is 94, estimating 4 additional prompts with refusal probability at or below 95%.
• Data scarcity shifts the posterior conservatively. With $n = 10$ generations, the model places more posterior mass on a smaller number of prompts having high refusal probabilities, since there is not enough data per prompt to believe they exceed the 95% threshold.
• At least one harmful prompt is almost never refused. The posterior distribution of $W_{\text{min}}$ indicates at least one prompt with a very low refusal probability, meaning some benchmark inputs are almost never refused despite being classified as harmful.
• Sequential sampling is framed as a matter of allocation. The paper argues the finite generation budget should be spent where it most reduces uncertainty about $W$, rather than spread uniformly across all prompts.

Methodology in Plain English

The setting is a "blackbox" system: a prompt goes in, text comes out, and the evaluator may not know what happens inside. The researchers fix a benchmark of $M$ prompts, generate $n_m$ stochastic outputs per prompt using the same decoding strategy the system would use in deployment, and have each output labeled with a binary judgment — for instance, refused versus not refused.

Statistically, they treat each prompt as having an unknown probability $\theta_m$ of triggering that binary behavior. A Beta prior is placed on each $\theta_m$; because a Beta prior paired with a binomial count gives a Beta posterior, the math stays simple and each prompt's posterior is $\text{Beta}(\alpha_m + r_m, \beta_m + n_m - r_m)$, where $r_m$ is the number of outputs with the behavior. The priors differ by case study: $\text{Beta}(1,1)$ (uniform) for pairwise preference, and $\text{Beta}(0.5, 0.5)$ for refusals, reflecting a weak prior expectation that prompts tend to be either almost always refused or almost always answered.

Uncertainty in the individual $\theta_m$ values then propagates into any aggregate metric $W = g(\theta_1, \ldots, \theta_M)$. For the metric that counts prompts above a threshold, the aggregate follows a Poisson binomial distribution, so its posterior is available in closed form. For the average-probability metric, the researchers draw 10,000 Monte Carlo samples from each $\theta_m$ posterior and average within each draw.

The second half maps evaluation onto a multi-armed bandit: each prompt is an arm, each generation is a pull, and the reward is information about $W$. The paper focuses on a reward derived from $W_{>\nu}$, whose variance is a sum of independent Bernoulli-type terms. Details beyond this point, and the sequential case-study results in Section 6, are not included in the provided text.

Why This Matters

• Impact on research: It pushes LLM evaluation from a single point estimate toward a full posterior distribution over behavioral metrics, and it foregrounds the stochastic nature of text generation as a distinct, quantifiable source of uncertainty — separate from sampling variability across prompts.
• Real-world applications:
  • Model selection and procurement: An organization choosing among blackbox API models can compare them with uncertainty ranges rather than a single leaderboard number.
  • Safety auditing and red-teaming: Regulators or internal auditors can report how many harmful prompts a model refuses with high probability, rather than how many it happened to refuse in one deterministic run.
  • Over-refusal and false-refusal monitoring: The same binary framework applies to whether a model wrongly declines benign prompts, using benchmarks that pair safe and unsafe items.
  • Privacy and information-leakage testing: The method applies to prompts probing information a model should have "unlearned," labeled by whether the output leaks sensitive content.
• Industry relevance: Because the method treats the system as a blackbox and only requires text in and text out, it is directly usable against commercial APIs. The sequential allocation component targets the practical constraint that every additional API call costs money and compute, making budget-aware evaluation a business-relevant design choice.

Future Directions

• Extending beyond deterministic judges. The paper explicitly treats the binary judge as deterministic for simplicity and notes that extensions to stochastic judges are beyond its scope — an obvious open direction.
• Generalizing the reward and aggregation functions. The sequential algorithms are derived with a reward based on $W_{>\nu}$, and the paper states the algorithms could be extended to other forms of $W$; the truncated text leaves the full sequential algorithm design and its case-study results unreported in the available content.
• Going beyond binary outcomes. The framing is binary throughout; the related work the paper cites includes Bayesian treatment of categorical evaluation metrics, suggesting a path to richer label sets.
• Scaling to more complex systems. The paper notes that LLM-based systems increasingly include extra logic, tool use, and agentic workflows, and keeps its approach agnostic to those internals — raising the question of how such internal structure interacts with the behavioral uncertainty being modeled.

Target Audience

Applied statisticians and NLP evaluation researchers who need to report uncertainty around behavioral metrics; safety, policy, and audit teams comparing or monitoring blackbox LLM deployments; and practitioners who run benchmark evaluations through paid APIs and want to spend their generation budget where it reduces uncertainty the most. Readers need only basic probability and Bayesian conjugacy to follow the modeling, though the LLM background sections assume no prior exposure to tokenization or decoding.

Authors’ abstract

It is increasingly important to evaluate how text generation systems based on large language models (LLMs) behave, such as their tendency to produce harmful output or their sensitivity to adversarial inputs. Such evaluations often rely on a curated benchmark set of input prompts provided to the LLM, where the output for each prompt may be assessed in a binary fashion (e.g., harmful/non-harmful or does not leak/leaks sensitive information), and the aggregation of binary scores is used to evaluate the LLM. However, existing approaches to evaluation often neglect statistical uncertainty quantification. With an applied statistics audience in mind, we provide background on LLM text generation and evaluation, and then describe a Bayesian approach for quantifying uncertainty in binary evaluation metrics. We focus in particular on uncertainty that is induced by the probabilistic text generation strategies typically deployed in LLM-based systems. We present two case studies applying this approach: 1) evaluating refusal rates on a benchmark of adversarial inputs designed to elicit harmful responses, and 2) evaluating pairwise preferences of one LLM over another on a benchmark of open-ended interactive dialogue examples. We demonstrate how the Bayesian approach can provide useful uncertainty quantification about the behavior of LLM-based systems.