TokenPowerBench: Benchmarking the Power Consumption of LLM Inference

AI summary

Overview

Research area: Machine learning systems and sustainable computing — specifically, power and energy benchmarking for large language model (LLM) inference.

Technical level: Intermediate. The paper assumes familiarity with LLM inference concepts such as prefill and decode phases, batch size, quantization, and tensor/pipeline parallelism, but explains the measurement methodology without requiring deep hardware expertise.

Scope: The paper introduces TokenPowerBench, an open-source benchmark that measures the power consumption of LLM inference across model families, hardware scales, and inference configurations, and applies it to models ranging from 1 billion to 405 billion parameters.

What This Paper Is About

Serving LLMs now dominates the energy footprint of AI: the paper cites an AWS report stating that inference consumes more than 90% of the energy consumption in the operational lifecycle of LLMs, yet existing benchmarks either target training, treat LLM inference as a generic machine learning task, or require expensive external power meters. The goal of this work is to provide a lightweight, extensible, open-source benchmark that measures GPU-, node-, and system-level power without specialized metering hardware and attributes energy specifically to the prefill and decode stages of each request. The authors use it to systematically study how batch size, context length, parallelism strategy, and quantization change joules per token across the Llama, Falcon, Qwen, and Mistral model series.

Key Contributions

1. First comprehensive benchmark: TokenPowerBench is described as the first open-source framework that couples phase-aware power telemetry with token-level normalization, filling a gap left by MLPerf and prior profiling tools. Table 1 in the paper claims it is the only listed approach that simultaneously provides node-level power, system-level power, LLM-specific coverage, hardware flexibility, cost analysis, real-world scenarios, standardized source, and an open benchmark.
2. Broad model coverage: The initial release profiles 15+ popular open-source LLMs from 1B to 405B parameters, spanning the LLaMA series, Mixtral series, Falcon series, and Qwen series, including both dense decoder-only transformers and Mixture-of-Experts (MoE) architectures.
3. First parameter-sensitivity analysis: Fine-grained comparisons of how inference parameters — batch size, context length, and quantization — impact energy consumption, with particular attention to the decode phase.
4. Frontier-scale case study: A detailed analysis built around Llama 3.1 405B as a representative state-of-the-art model, including FP16 versus FP8 comparisons and parallelism configurations.

Main Findings

• Super-linear scaling in energy with model size: Within the LLaMA-3 family, moving from 1B to 70B parameters increases energy per token by 7.3 times, even though parameter count grows 70 times. The authors attribute this to cache-bandwidth and memory-traffic penalties beyond pure FLOPs.
• Dense versus MoE efficiency: Mixtral-8×7B consumes roughly the same energy per token as a dense 8B model while delivering quality closer to a 56B dense model. Sparse routing that activates only two experts per token cuts token energy by 2–3 times compared with dense models of similar emergent accuracy.
• Engine impact is large and two-sided: Across all models tested, TensorRT-LLM and vLLM consume 3 times more than DeepSpeed and Transformers in the prefill stage, but they reduce energy per token by 25–40% relative to the Transformers engine. DeepSpeed-Inference sits between the two extremes, which the authors present as evidence that software optimization alone can rival architecture-level gains.
• Batch size lowers energy per token up to a point: Across six model families with batch sizes from 32 to 1024, the steepest drop occurs between 32 and 256, where most GPUs move from under 50% to nearly full utilization; the 70B model cuts per-token energy by about 25% in that range. Beyond batch 256 the curve flattens, with an overall two-to-three-fold spread between the smallest and largest batches.
• Longer context raises energy per token: Across ten models at prompt lengths of 0–2K, 2K–5K, and 5K–10K tokens, energy grows steadily because the prefill stage must process every input token while the cost of each new output token stays the same. For Llama3 70B, moving from 2K to 10K tokens raises energy per token by roughly a factor of three. The GPU-only trace and node-level trace show nearly identical patterns, confirming the accelerators dominate the added draw.
• Pure tensor parallelism is the most energy-efficient split: On three state-of-the-art models (Llama 3 405B, Falcon 180B, and Qwen 480B) across 16 H100 GPUs, TP16/PP1 beat TP4/PP4 and TP8/PP2 in every workload because long pipelines leave some GPUs idle. The gap between the best and worst split widens from about 40 J/token under Standard Load to more than 60 J/token under High Throughput.
• FP8 quantization cuts energy substantially: For Llama 3 405B, FP8 weight quantization versus FP16 cuts energy per token by roughly 30% across all three workload profiles. Measured per batch, total energy falls from about 45 kJ to 32 kJ under the heaviest load. The paper also reports that FP16 increases effective memory-bandwidth utilization by 13–17 percentage points and that FP8 raises end-to-end throughput, with the largest batch setting reported in the range of about 48–63 tokens/s; the wording of this sentence in the paper is ambiguous.
• GPUs dominate the energy breakdown: The paper states that GPUs typically account for over 60% of total energy use, while fans contribute only a few percent.

Methodology in Plain English

The authors built a three-layer tool: configuration, execution and measurement, and report generation.

Configuration. Users pick a model, a prompt dataset, and an inference engine through declarative options. The model pool covers decoder-only transformers (the LLaMA series) and MoE models (Mixtral), from under 1B parameters up to Llama 3-405B, which the paper notes needs at least 780 GB of FP16 memory and therefore multi-GPU or multi-node deployment. Prompt sets include Alpaca, with 52,002 prompts generated by OpenAI's text-davinci-003 engine, and LongBench, which has longer prompts with contexts up to 10k tokens; custom CSV or JSON prompts are also allowed. The supported engines are vLLM, TensorRT-LLM, DeepSpeed, and Transformers, with Ray used to launch distributed inference across nodes.

Measurement. During each run the tool samples power from GPUs via NVML/DCGM, from CPU and DRAM via Intel RAPL, and from the full node via IPMI or a rack-mounted PDU, plus network interface and fan tray sensors when available. Every sample carries the same timestamp and is tagged with the inference stage that was active — prefill or decode. The design rests on two decompositions: total energy equals prefill energy plus decode energy, and also equals GPU plus CPU plus DRAM plus other contributions.

Reporting. Results are integrated into metrics such as energy per token, energy per response, energy per second, peak power, and prefill energy, and written out as CSV and JSON. If the user supplies an electricity price and a regional carbon factor, the same pass converts kilowatt-hours into dollar cost and CO2 equivalents.

Experiments. The evaluation ran on an 8-node GPU cluster, each node equipped with 4 NVIDIA H100 GPUs (94 GB memory each), two Intel Xeon Gold 6426Y CPUs (16 cores, 32 threads each), and 512 GB of RAM. Models tested included Llama3 1B, 3B, 8B, 70B, and 405B; Mistral 7B, 24B, 8×7B, and 8×22B; Qwen 8B, 32B, and 480B; and Falcon 7B, 10B, and 180B. The main cross-model comparison covers 10 open-source models served by four engines on a single node with 4 H100 GPUs. The parallelism study used three SOTA models on 16 H100 GPUs under three workload profiles — Standard Load, High Concurrency, and High Throughput — with parallel splits of TP4/PP4, TP8/PP2, and TP16/PP1.

Why This Matters

Impact on research. The paper argues that existing benchmarks leave a gap: MLPerf Inference standardizes performance but does not systematically capture end-to-end power or normalize energy to LLM-specific service units; MLPerf Power emphasizes single-node setups, modest model sizes relative to frontier SOTA, and costly high-precision metering; LLM-Inference-Bench limits power analysis to accelerators themselves; and other work focuses on training-phase emissions. TokenPowerBench offers a reproducible, meter-free alternative that produces comparable measurements across days and clusters, which supports tracking regressions or demonstrating savings after an optimization.

Real-world applications:

• Data center operators can measure and forecast the operating expenditure of LLM serving, since the paper notes that for an LLM service electricity expenditure is the single largest component of ongoing operating cost.
• Sustainability teams can convert energy measurements into CO2 equivalents using a regional carbon factor, supporting reporting against sustainability targets.
• Practitioners tuning deployments can use the parameter sweeps to choose batch sizes, context limits, quantization levels, and parallelism strategies that lower joules per token.
• Researchers and national laboratories operating shared GPU testbeds can profile inference workloads across models and hardware generations without buying specialized power meters.

Industry relevance. The paper cites a global LLM market valued at approximately $5.6 billion in 2024, projected to exceed $35 billion by 2030 at a compound annual growth rate of 36.9%, and an AI inference market forecast to grow from $106 billion in 2025 to over $250 billion by 2030 at 19.2% CAGR. It also cites a Gartner prediction that by 2028 over 80% of data center workload accelerators will be dedicated to inference, a shift away from historically training-centric deployments. The authors position TokenPowerBench not as a replacement for MLPerf Power but as a complementary, agile tool answering a different question: what is the real-world energy cost of running a specific massive distributed model with a specific configuration on available hardware.

Future Directions

• Broader hardware coverage: The authors plan to extend beyond the current H100 testbed to other GPU architectures, including next NVIDIA generations, AMD accelerators, and emerging AI chips and DPUs.
• Accuracy–energy trade-offs: They intend to quantify the trade-off between inference accuracy and energy efficiency, giving users guidance on where energy savings begin to erode model quality. The current quantization case study reports no noticeable accuracy loss on the tested prompt set, but does not map a full accuracy–energy curve.
• Multi-node and cluster-wide characterization: The paper notes that cluster-wide energy variation and deployment modes from single-node to multi-node distributed inference remain an area existing work does not systematically explore, and its own broad model sweep is presented on a single node while the SOTA parallelism study uses 16 GPUs.
• Evolving the benchmark with the field: The authors state the landscape of LLM inference is rapidly evolving and that TokenPowerBench is designed to evolve with it, though specific roadmap items beyond the above are not enumerated.

Target Audience

This paper is most useful to systems and performance researchers studying LLM inference efficiency, data center and infrastructure operators responsible for GPU-backed inference services, and sustainability or capacity-planning teams that need energy and cost figures for LLM deployments. Practitioners tuning inference engines — choosing batch sizes, context lengths, quantization formats, or tensor versus pipeline parallelism — will also find the parameter sensitivity results directly actionable. Readers need a working knowledge of LLM serving terminology, but not of power instrumentation.

Authors’ abstract

Large language model (LLM) services now answer billions of queries per day, and industry reports show that inference, not training, accounts for more than 90% of total power consumption. However, existing benchmarks focus on either training/fine-tuning or performance of inference and provide little support for power consumption measurement and analysis of inference. We introduce TokenPowerBench, the first lightweight and extensible benchmark designed for LLM-inference power consumption studies. The benchmark combines (i) a declarative configuration interface covering model choice, prompt set, and inference engine, (ii) a measurement layer that captures GPU-, node-, and system-level power without specialized power meters, and (iii) a phase-aligned metrics pipeline that attributes energy to the prefill and decode stages of every request. These elements make it straight-forward to explore the power consumed by an LLM inference run; furthermore, by varying batch size, context length, parallelism strategy and quantization, users can quickly assess how each setting affects joules per token and other energy-efficiency metrics. We evaluate TokenPowerBench on four of the most widely used model series (Llama, Falcon, Qwen, and Mistral). Our experiments cover from 1 billion parameters up to the frontier-scale Llama3-405B model. Furthermore, we release TokenPowerBench as open source to help users to measure power consumption, forecast operating expenses, and meet sustainability targets when deploying LLM services.