The Pulse

Epoch AI introduces index for measuring inference capacity

Epoch AI has published an index that compares AI chips’ modeled inference capacity using memory and communication as well as compute. Its report estimates that an Nvidia GB300 can serve about 25 times as many agentic inference requests per

AI.info Team · Written with AI · fact-checked · 2026-11-09

A broader way to compare chips

Epoch AI has published a report introducing an index intended to compare AI chips’ inference capacity using memory and communication limits alongside raw computing speed. The report, written by Jason Li, says comparisons based only on floating-point operations per second can miss large differences in how many requests a chip can serve.

Epoch AI estimates that an Nvidia GB300 provides about 25 times the inference capacity per chip of an Nvidia H100 on the agentic workloads it modeled, despite having only 2.5 times the 8-bit FLOP per second. The report says a FLOP/s-only comparison would understate the GB300’s relative capacity by a factor of ten, because memory access and inter-chip communication can constrain inference more than arithmetic throughput.

The Inference Performance Index, or IPI, measures how many requests a chip can serve simultaneously at different required output speeds. Its first version focuses on agentic workloads and combines estimates for four representative models. Scores are given in GB300 equivalents, with the GB300 set to one; a score of two means Epoch AI estimates a chip could serve twice as many users or agents at once, averaged across the index’s workloads and speed requirements.

How the index is calculated

Epoch AI uses Nvidia’s AISimulate to estimate performance from model operations and hardware characteristics. These include arithmetic throughput, high-bandwidth memory capacity and bandwidth, and the bandwidth, grouping and latency of chip interconnects. The resulting figures are analytical projections of optimal performance under the modeled hardware limits, which the report calls speed-of-light estimates.

The index considers both prefill, which processes input, and decode, which generates output tokens. Epoch AI models the two phases separately and provisions enough prefill chips to avoid bottlenecking decode, then calculates capacity against the combined prefill and decode chips. During decode, larger request batches can raise total throughput while lowering the generation speed for each request.

To compare chips across different operating requirements, Epoch AI measures capacity from 50 to 400 tokens per second per request. It calculates the area under each capacity curve on a logarithmic speed scale, giving equal weight to each doubling of output speed, then divides each chip’s area by the GB300’s for the same model. The final score is a simple average of the four model-specific ratios.

Scores and comparisons

In the reported results, GB300 NVL72 scores one GB300 equivalent, compared with 0.12 for H200 HGX, which Epoch AI describes as roughly an eightfold difference in modeled capacity per GPU. Within Blackwell Ultra systems, GB300 NVL72 scores about 1.9 times B300 HGX, which scores 0.54. The report gives MI355X a score of about 0.55 and VR200 a score of 2.18.

The report also compares its estimates with SemiAnalysis InferenceX benchmarks for Kimi K3, which it says show a similar hardware ordering. For Kimi K3 alone, Epoch AI’s model predicts an approximately 46-fold GB300 advantage over H200, compared with an eightfold difference across the full model basket. It notes that InferenceX attributes part of Hopper’s weaker Kimi K3 results to missing software optimizations.

Limits and planned updates

Epoch AI says the scores describe modeled hardware potential and do not account for differences in software maturity, which can affect how much of that potential is achieved. The report also says it has not established that its numerical-precision choices preserve equivalent model quality, and identifies its assumed average of three accepted output tokens per decoding cycle as a modeling assumption rather than a measured rate for every model.

Epoch AI says future versions will update the workload and model selection as inference demand changes, and that it plans to add open models when their architectures can be modeled accurately with AISimulate. It advises comparing chips using scores from the same IPI version, since changing the model basket can change a chip’s score even if its hardware is unchanged.