The Pulse
Epoch AI Switches Capabilities Index to a Bayesian Model With Better-Justified Uncertainty Estimates
Epoch AI has updated the modeling behind its Epoch Capabilities Index with a Bayesian model. The model provides better-justified uncertainty estimates and lets Epoch AI calculate how likely one model is to be more capable than another.
AI.info Team · Written with AI · fact-checked ·
Epoch AI has updated the modeling behind its Epoch Capabilities Index. The Bayesian model produces probability distributions for model ECI scores, which Epoch uses to take uncertainty estimates directly from the model fitting and assess how likely one model is to be more capable than another.
How the ECI turns benchmark scores into capability estimates
The Epoch Capabilities Index, or ECI, uses Item Response Theory, a framework originally developed for educational testing that enables comparisons between students even when they took different exams with varying difficulty. Epoch adapts the framework to AI benchmarks by treating a model’s score as the expected share of questions it gets right. The ECI uses only benchmarks scored between zero and 100 percent.
Epoch models each benchmark’s difficulty and slope, which sets how quickly scores change as models become more capable. It assumes that a model’s performance on every benchmark depends on a single capability dimension, and, before fitting, rescales scores so random guessing maps to zero and the highest achievable score maps to one. Scores below chance are set to zero, while scores above the ceiling are set to one.
Too few scores per model for bootstrapping to be justified
The previous frequentist methodology fit the ECI by finding parameter values that minimized the squared error of the predicted score against the observed benchmark scores. Epoch estimated uncertainty with 500 bootstrap samples and reported 90 percent confidence intervals, resampling each model’s benchmark results with replacement before refitting the model. The median model had ten benchmark scores, while some models had as few as four.
With so few scores per model, bootstrapping shifts a model’s ECI considerably: a resample leaves out about a third of a model’s scores on average and counts others more than once. The new Bayesian model instead draws samples from the posterior using Markov Chain Monte Carlo. Each sample represents a plausible set of every model’s capability and every benchmark’s difficulty, slope and noise; taken together, the samples approximate a probability distribution for each model’s ECI.
The beta likelihood keeps probability between zero and one
The Bayesian model replaces the previous method’s constant-noise assumption with a beta likelihood. A normal likelihood can assign probability to scores below zero or above one, while the beta distribution keeps all probability between zero and one. The beta model’s noise shrinks as predicted scores approach either endpoint.
Each benchmark gets its own noise parameter, allowing the model to account for differences in how closely observed results tend to track predictions. For example, a model predicted to score 50 percent typically lands within 46–54 percent on METR Time Horizons, taken as a percentage score on the task suite. On VPCT, it typically lands within 32–68 percent.
Fixing the scale with the full set of benchmarks
After fitting the model on an internal raw scale, Epoch converts every sample to the ECI scale with a linear transformation that sets Claude 3.5 Sonnet to 130 and GPT-5 to 150. These anchors let Epoch compare results over time; the anchors themselves have no uncertainty. The 90 percent credible intervals run from the fifth to the 95th percentile of each model’s estimated distribution.
Epoch uses flat priors for model capability and benchmark difficulty, alongside loose hierarchical priors for benchmark slopes and noise. The capability and difficulty priors span −50 to 50 on the raw scale; with the current fit, that range converts to ECI values from −390 to +670. Models in the dataset sit between about 45 and 169.