The Pulse
MLPerf Adds RAG and Edge-Agent Tests to v6.1 Results
MLCommons’ MLPerf Inference v6.1 adds benchmarks for retrieval-augmented generation pipelines and edge-based agentic inference.
AI.info Team ·
MLCommons is expanding MLPerf beyond single-model speed tests, adding benchmarks for the multi-stage systems that now handle retrieval-augmented generation and on-device agents. The organization’s Inference v6.1 results, published September 16, introduce an end-to-end RAG test and an Edge Agentic Inference test while drawing submissions from a record 30 organizations.
The change addresses a mismatch that has widened as AI deployments have become more complicated. A conventional language-model benchmark can measure token generation, but it does not capture the embedding, retrieval, reranking and answer-generation steps in a RAG pipeline, or the growing context and repeated tool calls in an agentic coding session. MLCommons is now measuring those workflows as connected systems rather than isolated model calls.
MLCommons said the release includes the first peer-reviewed performance results for several recently released or upcoming platforms and shows gains of as much as 5.7 times compared with earlier benchmark rounds.
RAG Gets a Full Pipeline Test
The new End-to-End Retrieval-Augmented Generation benchmark measures two related workloads. The first ingests a document corpus and builds a vector database. The second answers questions against that database through an iterative pipeline that retrieves candidate passages, reranks them and sends evidence to one or more large language models.
That design makes the test materially different from a language-generation benchmark. Performance depends on how the system coordinates several models and services, not only on how quickly one accelerator produces tokens. MLCommons introduced the benchmark in August, describing it as the first MLPerf inference test to score a complete RAG pipeline from database creation through multi-hop question answering.
“We added the End-to-end RAG test because it’s clear that query-answering has evolved beyond simply an LLM trained on a corpus; stakeholders need to understand the real-world performance of the types of multi-step, multi-component pipelines that are being built today.”
— Miro Hodak, MLPerf Inference working group co-chair
Edge Agents Face a Different Constraint
The Edge Agentic Inference test targets a separate problem: a single user running a multi-turn agent on a device with fixed memory, power and processing capacity. Its workload uses the Qwen3.6-27B model in quantized form and replays coding interactions in which each turn depends on the conversation and tool results that came before it.
MLCommons measures time to first token, time per output token and end-to-end latency for each turn. The test runs one request at a time rather than measuring the high-concurrency throughput favored by data-center systems. It also includes an accuracy gate based on the Berkeley Function Calling Leaderboard v4 and an inline correctness check for the recorded coding trajectories.
The edge specification fixes the served context at 32,000 tokens. That gives the benchmark a controlled way to measure the effects of long agentic histories without treating each device’s memory limit as a different test condition. MLCommons’ submission guidance describes a reference performance set of 20 conversations and 1,007 turns.
“Likewise, we added the Edge Agentic Inference test because complex inference systems with agentic properties are increasingly hosted on edge computing devices, creating a new set of performance challenges our customers face today.”
— Miro Hodak, MLPerf Inference working group co-chair
Speculative Decoding Moves Into More Tests
Inference v6.1 also adds support for speculative decoding in the interactive scenario for two benchmarks, including the GPT-OSS task. The technique uses a faster prediction step to propose multiple tokens before a larger model verifies them, allowing systems to reduce some of the delay associated with sequential generation.
The release does not treat that optimization as a separate headline score. Instead, it adds the capability to the existing benchmark framework so submitters can report performance using a method already deployed in production inference systems.
Thirty Organizations Submit Results
MLPerf Inference v6.1 receives submissions from 30 organizations, including AMD, Cisco, CoreWeave, Dell, Google, Intel, Microsoft Azure, NVIDIA, Oracle, Red Hat and Supermicro. MLCommons identifies six first-time submitters: Atlas Inference, Crusoe, Orrick Industries LLC, ScitiX, VibeHPC and individual contributor Naeem Khoshnevis.
The round includes five new processors or accelerators: AMD Ryzen AI Max+ 395, AMD Instinct MI350P, Intel Arc Pro B70, NVIDIA Rubin and NVIDIA Vera Rubin NVL72. The last two appear in preview systems, while the AMD and Intel parts are listed as available. One submission uses 512 accelerators, the largest system submitted to MLPerf Inference so far.
MLCommons also reports two unusual heterogeneous configurations. One combines high-performance networking with accelerators from two vendors. The other distributes the system geographically across the Pacific Ocean, extending the benchmark beyond a single machine or data-center room.
Performance Gains Come With More Complicated Systems
Selected results show large gains over earlier rounds, but the comparisons depend on the workload and scenario. The best per-accelerator server result for the Visual Language Model test improves 2.99 times from v6.0, which was published six months earlier. The best per-accelerator server result for DeepSeek R1 improves 5.7 times compared with v5.1 from one year earlier.
Those figures measure the pace of hardware and software improvement inside particular MLPerf scenarios; they do not mean every AI deployment is 5.7 times faster. The more consequential change in v6.1 is the range of systems being measured: from individual edge accelerators running one interactive agent to large heterogeneous installations handling multi-model pipelines.
Frank Han, MLPerf Inference working group co-chair, said the results give customers more information about “the cost-benefit tradeoffs” involved in procuring and deploying AI systems. More than half of submitters used MLCommons’ new API-centric harness, which follows a client-server design over standard APIs and is intended to support the organization’s planned MLPerf Endpoints suite.
MLCommons says MLPerf Endpoints will eventually replace Inference for data-center benchmarking. For v6.1, the API harness already provides a bridge between the benchmark’s established result tables and the deployment pattern the organization says it plans to measure next.