The Pulse
MLPerf Endpoints tests live AI services under production-like loads
MLCommons’ MLPerf Endpoints v1.0 measures live API endpoints across workloads and concurrency levels. Submitters can release provisional results after automated checks, before peer review is complete.

Testing the full stack
MLPerf Endpoints v1.0 tests live API endpoints under production-like loads, measuring a system’s behavior across different levels of demand rather than giving it a single peak score. In a post published November 20, MLCommons said the benchmark covers the hardware, model, serving software, batching behavior and networking together.
The benchmark returns a set of measured operating points, or a Pareto frontier, showing throughput against per-user speed at different concurrency levels. The measurements include system throughput in tokens per second and the 90th-percentile time to first token.
MLCommons says the resulting LoadStair visualization uses discrete measurements and step functions, rather than interpolated curves, to show operational thresholds and potential performance cliffs. The organization says the data is not estimated or interpolated.
Workloads and models
The suite includes single-shot summarization using Llama 3.1 8B Instruct and GPT-OSS 120B, with CNN/DailyMail data and standard document pipeline traffic shapes. Its single-shot reasoning workload uses DeepSeek-R1 with hard math and science prompts that stress sustained token generation.
A multi-turn agentic coding workload uses Qwen 3.6, Kimi K3 and DeepSeek V4 Pro. It replays 613 trajectories with expanding context loops.
The organization says the suite uses workloads customers are using, rather than synthetic tests. Agentic benchmarks require seven discrete measurement points, while non-agentic suites require eight to 32 points, including a dedicated Offline run.
Preparing a submission
Submitters must sign the MLCommons Contributor License Agreement, obtain a PRISM API key and install the reference clients. They then run the inference-endpoint tool against a live endpoint at their selected concurrency levels.
The submission package must include system_desc.json, describing the hardware and software configuration, and system_power.json, stating provisioned system power for power-normalization metrics. Submitters validate the package locally before uploading it through the submission command-line tool, which generates an ID and opens a peer-review request.
The documentation says a non-agentic benchmark requires a minimum of eight points and five accuracy runs, with warmup in addition to the specified steady-state runs. Agentic benchmarks require seven points and four accuracy runs; points above concurrency 32 each run for 2,400 seconds.
The submitter documentation says the v1.0 rules are a working draft and that some sections governing run duration and disclosures are tentative or pending working-group ratification. It directs submitters to the policy repository when the documentation and policy differ.
Review and publication
Under the rolling submission model, results are published in cohorts on the first and third Wednesday of each month. A submission must pass automated compliance checks and peer review at least one business day before a cohort date to make that cohort.
MLCommons also permits provisional results to be shared after an automatic check, before peer review is complete. Final verified results can be held under embargo to coincide with an event or launch.
The submission documentation describes confidential review as the default publication mode. Provisional results are marked as pending peer review until finalization, while an embargo on a provisional submission also delays the start of review.
Sources
- Original source MLCommons · Mlperf endpoints how to submit mlcommons.org
Other pages read
- MLCommons · Endpoints submission docs docs.mlcommons.org
- MLCommons · How submission works docs.mlcommons.org
- MLCommons · Before you begin docs.mlcommons.org