The SIMPLE Law for Energy Cost of LLM Inference

AI summary

Overview

Research area: Energy-efficient AI systems — analytical modeling and prediction of the energy consumed by large language model (LLM) inference.

Technical level: Intermediate to Advanced. The paper derives closed-form FLOP and memory-operation counts from Transformer kernels, so familiarity with attention mechanisms (MHA, GQA, MQA, MLA) and roofline-style reasoning helps, though the central results are stated in plain architectural terms.

Scope (one sentence): The paper derives a four-parameter, architecture-grounded law — SIMPLE — that predicts per-request LLM inference energy from architectural hyperparameters and input/output lengths without per-model calibration, and uses it to explain how model size, attention grouping, reasoning budgets and agentic workloads change energy cost.

What This Paper Is About

Inference is where most of an LLM's life-cycle energy goes (the paper cites up to 90%), yet the industry bills, budgets and benchmarks in tokens without any analytical statement of what a token actually costs. Existing models either fit energy empirically as a function of input and output lengths, or — in the case of the most accurate prior model, SweetSpot — derive a quadratic form but absorb the architecture into fitted coefficients, so the number of layers, hidden dimension and attention organization never appear as terms and each new LLM requires its own calibration. The authors' goal is a law written in the architecture's own hyperparameters that accounts explicitly for the attention mechanism (MHA and GQA), generalizes to LLMs never benchmarked, and explains which hyperparameters set the cost of a token.

Key Contributions

1. A taxonomy of prior inference-energy models classified by functional form (linear, bilinear, power-law, quadratic), grounding theory (derived vs. empirical), whether they require per-LLM fitting, and whether they are attention-aware. The authors conclude that no prior model is simultaneously interpretable, LLM-agnostic, attention-aware, and applicable to the full prefill-plus-decode pipeline.

2. Closed-form FLOP and memory-operation (MemOP) expressions for LLM inference covering both MHA and GQA, derived separately for prefill and decode and parameterized by the KV-to-query head ratio R = n_kv/n_q, so that reduced KV-cache traffic under grouped-query attention appears explicitly rather than being assumed away.

3. The SIMPLE law (SIMPLE Inference Model to Predict LLMs Energy), a four-parameter model that separates prefill from decode and further splits decode into KV-cache cost and next-token (feed-forward) computation, selected from a family of analytically derived models of increasing granularity as the best accuracy-compactness trade-off.

4. Application of the law as an explanatory tool across LLM sizes, families and architectures, yielding derived results on hidden-dimension scaling, GQA grouping, the output-length energy minimum, reasoning-budget effects on Intelligence per Joule, and agentic serving costs with and without prefix caching.

Main Findings

• Accuracy without per-LLM calibration: Evaluated on traces of 13 dense LLMs from six families (OPT, LLaMA, Gemma, Falcon, Qwen2, Granite) on NVIDIA H100 GPUs, SIMPLE attains R² = 0.969 and 11.97% MAPE — 3.1–3.5 times below the state of the art — using four parameters and no per-LLM fitting.

• Hidden dimension is the dominant hyperparameter: Prefill energy scales quadratically with the hidden dimension d, while layer depth contributes only linearly, so larger LLMs suffer most from long inputs.

• Attention grouping has a large effect: More aggressive GQA KV grouping reduces per-token energy by up to 3 times relative to standard MHA, making the attention configuration an impactful architectural lever.

• Reasoning budgets overshoot the derived optimum: The law yields an analytical expression for the output length τ_out that minimizes per-token energy (the "sweet spot"); increasing the reasoning token budget pushes τ_out beyond that minimum and drives super-linear energy growth. Accuracy does grow with the budget, but energy grows faster, so Intelligence per Joule peaks at an intermediate budget and falls by up to 3.14 times between the lowest and highest budgets across a 4.0 times reasoning-budget range.

• Agentic serving is far more expensive: Energy per request rises by 13.0 times relative to non-agentic requests, driven by growth in both the prefill and KV-cache terms with input length.

• Prefix caching does not fix the KV-cache cost: Caching removes most of the prefill cost (−93%), but the KV-cache term is unaffected and accounts for 86% of the remaining energy, so even a cached agentic request still costs 4.8 times a non-agentic one.

• Positioning against prior models: SweetSpot is reported to reduce bilinear-model MAPE from over 30% to below 2%, but is LLM-specific and implicitly assumes MHA; the analytically grounded model of Vartziotis et al. is LLM-agnostic but reports roughly 20% mean relative error with hard-to-interpret expressions; Niu et al. is attention-aware but covers only the decode phase and requires measured runtime inputs.

Methodology in Plain English

The authors start from the standard decoder-only Transformer and count, on paper, the arithmetic operations (FLOPs) and memory movements (MemOPs) that a request actually requires — treating a matrix multiplication as 2mnk FLOPs and mk + kn + mn memory operations.

They do this twice: once for the prefill phase, where the whole input of length τ_in is processed in parallel, and once for decode, where one token is produced per step and the KV-cache grows by one entry each step. Summing the per-step decode cost over all generated tokens uses the arithmetic-series identity, which is what produces the quadratic terms in the final expression.

To make the derivation cover modern architectures rather than just vanilla multi-head attention, they introduce the ratio R between KV heads and query heads, so key and value weight matrices have shape d × d·R. Setting R = 1 recovers standard MHA, and smaller R captures GQA. They neglect biases, layer normalization, softmax and dropout, fix the FFN expansion factor at 4×, assume no weight or KV-cache quantization, and hold batch size constant.

The resulting complexity expressions are then grouped in different ways — the paper describes an "OPs-centric" family in which coefficients take the meaning of Joules per FLOP or Joules per MemOP, and at least one other grouping — yielding a family of analytical models of increasing granularity. The four-parameter SIMPLE model is the member of that family with the best trade-off between accuracy and compactness, and it is validated against TensorRT-LLM traces from the 13 H100-served LLMs. Normalization is to energy per token (E_tot / τ_out) rather than total energy, to isolate generation cost from input-processing cost.

Why This Matters

Impact on research. The paper shifts energy modeling from fitting to derivation: architectural hyperparameters appear as terms rather than being folded into coefficients, which means an architecture can be evaluated before it is built or served. It also supplies an attention-aware treatment to a literature that largely assumed MHA, and provides an analytically derived optimum output length that reasoning-budget and serving research can reason about directly.

Real-world applications:

• Architecture selection before deployment: comparing candidate designs (depth vs. width, MHA vs. GQA grouping) by predicted energy per token rather than by measuring systems that already exist.
• Reasoning-budget policy: choosing how many reasoning tokens to spend per query, since Intelligence per Joule peaks at an intermediate budget rather than at the largest one.
• Agentic serving and capacity planning: estimating per-request energy for long-context agentic workflows, and seeing that prefix caching alone leaves a KV-cache-dominated residual cost (4.8 times non-agentic even when cached).
• Cost and pricing analysis: explaining how a model with a lower listed price per token can incur up to 28 times the total cost of a nominally more expensive one, since token prices ignore how context length and attention organization set production cost.

Industry relevance. Inference is billed, budgeted and provisioned in tokens per second and tokens per joule, and the paper notes that median prompt lengths have moved from 750–1050 tokens to 68K tokens with agentic workflows. A calibration-free law usable on unseen LLMs lets operators forecast energy for hardware and models they have not yet measured, which matters for datacenter capacity, sustainability reporting, and the "Intelligence per Watt" style metrics the paper references.

Future Directions

• Extending beyond MHA and GQA: the derivation covers those two mechanisms, and MLA is discussed as a GQA variant rather than being derived explicitly; a fully derived MLA treatment is an open step.
• Removing fixed modeling assumptions: the law assumes a constant batch size, no weight or KV-cache quantization, and a 4× FFN expansion factor (which some models in the evaluation set deviate from). Batch-aware and quantized variants are natural extensions, particularly since batch size is commonly modeled in distributed-inference work but not in single-GPU models like this one.
• Folding hardware and multi-GPU effects in: the validation is on NVIDIA H100 GPUs; whether the four parameters transfer across hardware generations and multi-node serving, as opposed to being hardware-conditional, is not established in the reported content.
• Using the derived optimum operationally: the analytically derived energy-minimizing output length and the Intelligence per Joule peak invite scheduling and budget-allocation policies that stop reasoning at the energy-optimal point rather than the accuracy-optimal one — the paper establishes the curve but does not report a deployed policy.

Target Audience

Researchers and practitioners in efficient AI and ML systems who need to predict or reason about LLM inference energy without running a measurement campaign — particularly those working on serving infrastructure, accelerator and datacenter capacity planning, sustainability reporting for generative AI, reasoning-budget and test-time-compute policy, and architecture design. It is also relevant to readers benchmarking energy models, who will find the taxonomy of linear, bilinear, power-law and quadratic formulations useful as a map of the field.

Authors’ abstract

Generative AI is billed per token, yet tokens are not equally expensive. Each token enters the KV-cache and raises the cost of those that follow, so energy grows non-linearly along a sequence whose tokens look identical. No analytical model states what a token costs, and the most accurate SOTA model is a fit, not a law: it absorbs architecture into fitted coefficients, requiring per-LLM fitting. We derive closed-form expressions of the computational and memory-access complexity of LLM inference in terms of architectural hyperparameters, covering MHA and GQA, and compare a family of analytical models of increasing granularity. From these, we obtain the SIMPLE Inference Model to Predict LLMs Energy (SIMPLE), a four-parameter law that separates prefill from decode, and decode into KV-cache cost and next-token computation. Over traces of 13 dense LLMs from six families on NVIDIA H100 GPUs, it attains R2 = 0.969 and 11.97% MAPE, 3.1-3.5x below the SOTA, without per-LLM calibration. The SIMPLE law explains as well as predicts: prefill energy grows quadratically with the hidden dimension, so larger LLMs suffer most from long inputs; and more aggressive GQA KV grouping reduces the per-token energy by up to 3x relative to standard MHA. Increasing the reasoning token budget pushes output length beyond the analytically derived energy minimum, driving super-linear energy growth; although accuracy grows with the budget, energy grows faster, so Intelligence per Joule peaks at an intermediate budget and falls by up to 3.14x across a 4.0x reasoning-budget range. Agentic serving raises energy per request by 13.0x relative to non-agentic ones, driven by the growth of both the prefill and the KV-cache terms with input length; while prefix caching removes most of the prefill cost (-93%), the KV-cache term is unaffected, so even a cached agentic request costs 4.8x more.