The Pulse
NVIDIA Replaces GenAI-Perf With AIPerf for LLM Benchmarking
NVIDIA presents AIPerf as the successor to GenAI-Perf, with multiprocess load generation, broad workload support and metrics for testing LLM inference systems.
AI.info Team ·
NVIDIA Draws a Line Under GenAI-Perf
NVIDIA has designated AIPerf as the successor to GenAI-Perf, describing the new tool as a ground-up rewrite for benchmarking large language model inference. The company announced the change in a technical post published September 18, 2026.
According to NVIDIA, conventional benchmark clients can become a limiting factor when engineers test inference servers at high concurrency or request rates. Single-process tools may run into Python’s Global Interpreter Lock, making it harder to determine whether measured performance reflects the server or the client generating the workload.
AIPerf uses a multiprocess architecture. Worker processes generate traffic, while separate record-processing services handle returned measurements. The components are coordinated over ZeroMQ, a design NVIDIA says helps prevent the benchmark client from becoming a bottleneck during server testing.
Multiprocess Load Generation Targets the Client Bottleneck
The tool supports more than 15 endpoint types, including chat, responses, NIM rankings and image generation. It also supports public datasets such as ShareGPT and trace-replay formats from Mooncake, Baseten and WEKA AgentX.
That range is intended to cover both synthetic tests and workloads based on captured production traffic. NVIDIA presents AIPerf as a single client for quick smoke tests, custom datasets and more realistic replay scenarios.
AIPerf also provides several ways to shape request traffic. It supports constant, Poisson and gamma arrival patterns, along with tunable burstiness and gradual ramps for concurrency and request rate. Synthetic distributions can also vary input and output sequence lengths, including vLLM and SGLang range-ratio modes for variable input-sequence and output-sequence lengths.
A First Benchmark With vLLM
NVIDIA’s walkthrough uses Qwen3-0.6B served through vLLM. The model is small enough to run on a single GPU and is used to demonstrate the measurement workflow rather than to establish a model-specific performance result.
NVIDIA installs the client with uv tool install aiperf or inside a virtual environment. The example then runs a streaming chat benchmark with fixed 128-token input and output lengths:
aiperf profile \ --model Qwen/Qwen3-0.6B \ --endpoint-type chat \ --streaming \ --url localhost:8000 \ --synthetic-input-tokens-mean 128 \ --synthetic-input-tokens-stddev 0 \ --output-tokens-mean 128 \ --output-tokens-stddev 0 \ --extra-inputs min_tokens:128 \ --extra-inputs ignore_eos:trueThe post says the additional flags tell the model to emit 128 tokens rather than stopping early. Without them, the output token count is only a suggestion and the model may stop well short of the target.
Streaming is required to measure time to first token and inter-token latency. Without streaming, the server sends the completed response as a whole, leaving no first-token or decode-token events for the client to measure.
Metrics and GPU Telemetry
AIPerf reports time to first token, inter-token latency, request latency and output-token throughput. The metrics include averages, minimums, maximums, standard deviations and percentile breakdowns such as p50, p90, p95 and p99.
The tool prints a metrics table to the console and writes full results to CSV and JSON. Figure 2 shows the command line needed to repeat the benchmark and the locations of the output files.
When DCGM or pynvml is available, AIPerf also collects GPU power draw, utilization and memory consumption. NVIDIA says putting those measurements alongside latency results can help identify whether a latency increase is associated with memory pressure or another device-level event.
Poisson Traffic Shows the Effect of Contention
NVIDIA’s second example sends an average of 10 requests per second using a Poisson arrival pattern. It uses a mean input length of 512 tokens with a standard deviation of 128, a mean output length of 128 tokens with a standard deviation of 32, and a fixed random seed for reproducibility.
The Poisson pattern produces gaps and bursts rather than a perfectly regular stream. NVIDIA reports input lengths ranging from 154 to 818 tokens in the example and says the workload creates a wider distribution of time to first token than a single-user run.
The difference reflects several sources of contention: more requests compete for GPU access, input lengths vary during prefill, and prefill and decode operations overlap. The single-concurrency case produces lower time to first token but does so at the cost of throughput.
Additional Workloads
NVIDIA says AIPerf also supports multi-node Kubernetes deployments, KV-cache reuse warm-up mechanics, production trace replay, prefix synthesis, custom datasets and sweep configurations across concurrency levels. The company directs new users to the AIPerf repository, tutorials and documentation for further configuration details.