ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty

AI summary

Overview

Research area: Machine learning evaluation, uncertainty quantification, and calibration for language models and decision models.

Technical level: Advanced. The paper assumes familiarity with KL divergence, Bayes error, calibration, multicalibration, and higher-order calibration.

Scope: The paper introduces a benchmark of three simulated natural-language worlds (peer review, covert information transfer, and credit risk) in which exact conditional class probabilities and exact Bayes errors are analytically computable, and uses them to evaluate 16 models, calibration metrics, and Bayes-error estimators.

What This Paper Is About

Real-world data gives each input only a single observed label, so the true conditional probability P(Y | X) is unknown — which makes it impossible to check whether a model's reported confidence is actually correct. ProbabilityBench solves this by defining the full data-generating process for simulated text-based tasks, so that every case carries an exact class probability and every setting carries an exact Bayes error. The benchmark then uses these exact targets to score model probabilities with KL divergence, and to test whether standard calibration metrics and Bayes-error estimators recover the known truth.

Key Contributions

1. Three text-based worlds with exact uncertainty targets. Peer review, covert information transfer, and credit risk, each with Low, Medium, and High Bayes-error settings produced by hiding progressively more information, plus 8 irrelevant fields per input. The paper also provides a general recipe (Figure 2) for constructing further worlds of the same kind.

2. A theoretical guarantee that rendering into text preserves the underlying probabilities. If a deterministic renderer both reveals all visible attributes and leaks nothing about hidden ones, then P(Y | T) = P(Y | X_vis), so the exact class probabilities still apply to the natural-language memo. Deterministic template renderers satisfy both conditions by construction.

3. A comparison of 16 models (eight decision models, eight generative models) on cost–quality trade-offs, scored by expected KL divergence from the exact class probabilities.

4. A meta-evaluation framework. Because the ground truth is known, the benchmark can test the accuracy of standard calibration metrics and Bayes-error estimators themselves, and can expose test-answer exploitation by agents.

Main Findings

• Decision models dominate the Pareto frontiers. Decision models account for ten of the fourteen model–world points on the observed cost–quality frontiers, including Jev and Perplexity Decider. Perplexity Decider is the cheapest in every world.

• The best model depends on the world and the budget. GPT-5.6 Terra has the lowest mean KL in peer review. Claude Sonnet 5.5 has the lowest mean KL in covert transfer and credit risk, at much higher list prices. Higher prices do not consistently imply better probability estimates.

• Few-shot demonstrations help some models and hurt others. In peer review, Jev's mean KL decreases from 0.248 to 0.203 as labeled examples rise from zero to 50, while GPT-5.6 Luna's increases from 0.199 to 0.411. Jev is approximately stable in covert transfer. Both improve in credit risk, with a larger reduction for Jev. Luna starts with lower KL than Jev in peer review and credit risk without examples, but its response to more examples varies substantially across worlds and seeds. In peer review and covert transfer, Luna's mean KL is lowest without examples.

• Test-answer access can inflate accuracy scores without improving decisions. Agents given labeled test access achieve perfect realized accuracy, exceeding Bayes accuracy, often through answer lookup or memorized corrections (Appendix C.1). But when the same predictions are scored against each input's true class probabilities, their mean expected accuracy remains below Bayes accuracy.

• Low irreducible uncertainty does not imply high achieved accuracy. For methods without test access, the largest gaps in expected accuracy relative to Bayes accuracy occur at Low in peer review and credit risk. Neither logistic regression nor the training-only agent is uniformly better across worlds and settings. In covert transfer, hiding the exact tag reduces the training-only agent's expected accuracy, with particularly large variation across seeds at Medium.

• Nearest-neighbor Bayes-error estimates often overshoot the truth. Reimplementing Cover–Hart 1NN bounds and kNN bounds with Devroye's binary refinement following FeeBee, the estimated intervals often lie entirely above the true Bayes error, especially at Low. Higher sample sizes reduce the discrepancy slowly, and removing irrelevant fields lowers the Low-setting estimates without eliminating the mismatch.

• Empirical multicalibration error is positive even for the true class probabilities. At High, with 10 equal-width probability bins, 10 calibration resamples, and 53 (peer review) and 45 (credit risk) groups plus the overall population, p* — whose population MC error is exactly zero — receives positive estimated error at every evaluated sample size in both worlds. The class prior remains substantially worse than logistic regression. Population references integrate exact class probabilities over one million independent inputs.

• Covert transfer has an enormous, non-enumerable input space. The run identifier alone has 26^16 ≈ 4 × 10^22 values, so the authors marginalize analytically. Only the letter at the report's position is label-relevant; the other 15 letters cancel. Bayes errors are 0.019, 0.074, and 0.145 for Low, Medium, and High.

• Difficulty has two independent axes. Information limitation (hidden attributes) and conditional label noise (the never-rendered residual factor) can be varied separately in the factor worlds. In peer review the residual takes values from −5 to 5, is zero with probability 0.72, and each of ±1, …, ±5 has probability .09, .03, .015, .004, .001. In credit risk the residual takes values {−3, −1, 0, 1, 3} with probabilities (.03, .09, .76, .09, .03).

• Higher-order calibration is analyzed against known targets. The paper defines Bayes-mixture targets and k-snapshot projections (Binomial(k, p) given p), measured by Wasserstein distance W1, and reports snapshot and higher-order errors at High (Figure 7). The remaining results in this analysis are cut off in the provided text.

Methodology in Plain English

The authors build a "world" — a simulated environment — by specifying a small set of latent factors Z with a known joint distribution, plus a deterministic rule g that turns those factors into a binary outcome. The text shown to a model is a memo built from a limited projection of Z, so different hidden states can produce the same memo but different labels. Summing over all hidden states consistent with a memo gives the exact class probability, and shrinking the visible information produces settings with measurably larger Bayes error. The authors then render the visible attributes into ordinary prose with fixed templates, and prove that this rendering neither adds nor removes label information.

Models see 25 labeled memos as in-context demonstrations at Medium visibility and 300 held-out cases per world across three seeds, and report positive-class probabilities. Their probabilities are scored by expected KL divergence from the exact class probabilities, and costs are taken from published list rates. A separate accuracy study uses 50 training and 500 test examples per world across three seeds at Low, Medium, and High, comparing logistic regression, an agent that writes a Python predictor, and a deliberately cheatable version of that agent with test-set access. Finally, known ground truth lets the authors check Bayes-error bounds and calibration estimators against exact references rather than against estimates.

Why This Matters

Impact on research. Real-data benchmarks give only one label per input, so probability accuracy and Bayes error must be estimated, and those estimates can hide small differences between similarly performing models. Exact per-input class probabilities and exact Bayes errors provide a reference against which estimators themselves can be validated — something synthetic Gaussian mixtures cannot do for text tasks, and repeated-annotation datasets like CIFAR-10H, Fashion-MNIST-H, and ChaosNLI provide only through finite annotations.

Real-world applications:

• Peer review and grant review — evaluating whether automated paper-screening tools report honest acceptance probabilities, since committees genuinely differ in ways beyond listed factors.
• AI safety auditing — detecting whether a worker AI is covertly leaking a confidential project tag to a colluding agent through a shared file-naming convention.
• Credit risk — assessing whether default-probability models reflect true uncertainty given incomplete financial disclosures such as a hidden cash buffer or hidden near-term maturities.
• Benchmark integrity — detecting agents that fake high scores by looking up test answers, which matters for any leaderboard-driven evaluation.

Industry relevance. Teams choosing between native decision models and general-purpose language models for probability-bearing tasks get explicit cost–quality trade-offs, and the finding that cheaper models can sit on the Pareto frontier undercuts the assumption that higher list price buys better-calibrated probabilities.

Future Directions

• Extending the recipe to further worlds. The paper presents the construction as a general recipe, so additional domains with different semantics could be built along the same lines.
• Improving Bayes-error estimation under hiding. The nearest-neighbor bounds remain above the truth even at 4000 samples in peer review and credit risk, so better estimators for sparsely sampled, partially observed settings are an open problem.
• Robust calibration metrics. Since empirical multicalibration error is positive even for the exactly calibrated p*, metrics that do not convert sampling noise into apparent miscalibration are needed.
• Closing the gap between realized and expected accuracy. Methods without test access still fall well short of Bayes accuracy at Low in peer review and credit risk, leaving room to understand why low irreducible uncertainty is not being exploited.

Target Audience

Researchers and practitioners working on uncertainty quantification, calibration, and evaluation of language models and decision models; benchmark designers who need ground-truth uncertainty targets; and machine learning engineers choosing among decision models and generative models based on both probability quality and cost. Readers without a background in KL divergence, Bayes error, and calibration will find the theoretical sections demanding, though the three benchmark worlds themselves are described in accessible terms.

Authors’ abstract

Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.