The Pulse
MLCommons Wins $3.5 Million to Build an AI Agent Benchmark
MLCommons will use a $3.5 million, two-year award to develop MLPerf Agents, a benchmark measuring the efficiency of complete AI agent workflows. Its first planned workloads cover software engineering, research and data analysis, and custome
AI.info Team · Written with AI ·
A $3.5 million, two-year award will fund MLCommons’ new benchmark for measuring how efficiently AI agents complete real tasks, the nonprofit consortium announced October 15. Called MLPerf Agents, the benchmark is designed to track an agent’s work from a delegated goal through to a verified result—not just the speed of the model generating its responses.
MLPerf Agents measures the whole workflow
An agent can make repeated model calls, use tools, retrieve information, carry an expanding context, recover from failures and check its own work. MLCommons says MLPerf Agents will account for those steps alongside the agent harness, model selection, orchestration, state management and validation. The aim is to measure the resources consumed across the full system rather than isolate model-serving performance.
The planned metrics include cost, energy, latency and throughput, measured for tasks that the system successfully completes. MLCommons says success checks will draw on methods shaped by its AI Risk and Responsibility working groups. Common workloads, tracing and measurement rules are intended to make results comparable across different system designs, while hidden validation data and review and audit procedures are meant to limit benchmark gaming.
Three workloads are planned for the first release
MLCommons identifies software engineering, research and data analysis, and customer service as the initial workload areas. Those categories would test systems against distinct forms of delegated work, from coding tasks to information gathering and customer interactions. The announcement does not give specific task sets, scoring thresholds or a release schedule.
The project also builds on benchmark work MLCommons already has underway. Its MLPerf Inference working group is developing tests for agentic inference, including multi-turn exchanges, tool calls, growing context and cache reuse. MLCommons’ separate MLPerf Client benchmark page describes agent tests for software engineering and data analyst tasks on personal computers, with end-to-end duration as a metric; MLPerf Agents is presented as a broader measure of the complete workflow.
Reddi says buyers need more than token-speed results
David Kanter, MLCommons founder and head of MLPerf, and Professor Vijay Janapa Reddi, co-founder and vice president of MLCommons, will lead the work with the consortium’s community. The announcement names Qualcomm, NVIDIA, Intel, AT&T, Workato and Gimlet Labs among the organizations supporting the effort.
“MLPerf Agents will give developers the evidence they need to find and fix those inefficiencies, and give buyers the evidence they need to choose efficient systems.”
Vijay Janapa Reddi, Board Member and Vice President of Research, MLCommons
The award comes through STRIDE Ventures’ AI Efficiency Challenge, a program supported by the U.S. National Science Foundation’s Directorate for Technology, Innovation and Partnerships and administered by Start2 Group under an agreement with NSF. The challenge solicitation calls for benchmarking teams to develop industry benchmarks for assessing AI software-stack efficiency and validating efficiency gains. For MLPerf Agents, the next concrete milestones are still unspecified: the announcement gives no publication date, workload specifications or rules for how a task’s successful completion will be judged.