The Pulse
WEKA-backed scorecard challenges GPU use as an AI inference metric
NAND Research has published a scorecard for measuring AI inference by completed work rather than GPU utilization. WEKA, which sponsored the research, and Nebius tested one of WEKA's products against an HBM-only configuration.

A new scorecard from NAND Research proposes judging AI inference systems by completed work rather than how busy their GPUs are. WEKA sponsored the research and supplied the workload used to test the framework, according to a post by its CEO, Liran Zvibel.
Published on November 16, the Inference Economics Scorecard sets out measures and testing rules for systems that run AI models. WEKA and cloud provider Nebius also reported a benchmark comparing WEKA's memory technology with a configuration that used GPU high-bandwidth memory alone.
Why utilization can mislead
WEKA argues that a GPU can remain busy without advancing an AI agent's task. As an agent works through multiple steps, it re-reads context from earlier in the session; if that context has been displaced from GPU memory, the system must rebuild it.
That recomputation can keep utilization high while repeating work already performed, the company says. Utilization also does not show whether requests finished, timed out or had to be retried.
The scorecard sets out nine measures: tokens per watt, tokens per rack, cost per inference, GPU goodput, time to first token and end-to-end latency, memory efficiency, operational density, agent concurrency, and context retention.
Its four testing rules call for the same workload and service targets across systems, measurement of the whole serving stack, runs of at least eight hours and a count of successful output only. The framework also calls for failures to be counted in the benchmark, according to a WEKA release.
A sustained comparison
WEKA and Nebius applied the rules to a test on an NVIDIA HGX B300 node with eight GPUs serving DeepSeek-V4-Pro. Each configuration ran for eight hours using sessions drawn from a corpus of more than 5,000 real agentic coding sessions; failed and timed-out requests were counted.
The workload kept 512 sessions active at a concurrency of 32, the benchmark report states. One configuration used high-bandwidth memory, or HBM, alone; the other used WEKA NeuralMesh with Augmented Memory Grid, which pools NVMe storage across a cluster as a shared context tier.
In results reported by the companies, the shared-tier configuration produced 3.7 times as many output tokens per GPU-hour as HBM-only serving on the same GPUs. Among successfully completed requests, 97% reached a first token within five seconds, compared with 44% for HBM-only serving.
The benchmark report says the HBM-only run recorded 533 errors, or 2.32% of requests, while the Augmented Memory Grid run recorded 359 errors, or 0.64%. Both runs used the same hardware and workload generator, but the latter attempted more requests during the equal test windows.
Limits and open tools
The comparison used HBM-only as its baseline because, at the time of testing, no DRAM offload implementation could handle DeepSeek V4, the benchmark authors say. They add that other solutions have matured since the test.
WEKA CEO Liran Zvibel says a shared context tier is unlikely to change results much when sessions are short, context is small or the working set fits comfortably in HBM. The reported gains appeared under long sessions, concurrent use and runs measured over hours.
WEKA has also published its kvlab corpus, capture router and replay tools. The company says its engineers captured more than 5,000 real sessions and over half a million requests by recording request structure rather than code or conversation content.
SemiAnalysis plans to share AgentX results on WEKA NeuralMesh as testing progresses. Its AgentX benchmark uses the same traces, and NVIDIA's AIPerf tool includes a loader for them.
Sources
Related stories
- AMD details GPU-initiated DMA transfers for multi-GPU workloads
- TurboPairFormer speeds OpenFold3 training up to 1.73× in tests
- Hyve to offer Astera Labs' Orion components in AI inference racks
- Cerebras puts split AI inference into production with AMD GPUs
- AMD reports faster inference glue kernels using AITER config pins