SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference

AI summary

Overview

Research area: Energy-efficiency modeling for Large Language Model inference, sitting at the intersection of AI systems, GPU hardware power measurement, and analytical performance modeling.

Technical level: Advanced. The paper derives closed-form expressions from the computational and memory-access complexity of Transformer prefill and decode, then validates them against hardware power measurements.

Scope: A single paper proposing and empirically validating an analytical model (SweetSpot) that predicts how LLM inference energy per output token depends non-linearly on input and output sequence lengths, and where peak efficiency lies.

What This Paper Is About

Existing methods treat the energy cost of LLM inference as a simple linear function of the number of input and output tokens, which implies that every generated token costs roughly the same. The authors argue this ignores the autoregressive structure of Transformers — with distinct prefill and decode phases that scale quadratically and linearly respectively — and therefore misses the existence of a generation energy minimum. Their goal is to derive an analytical model from the architecture's actual compute and memory-access complexity, and to test whether it correctly locates these efficiency "sweet spots" on real hardware.

Key Contributions

1. The SweetSpot analytical model. A formulation derived from the computational and memory-access complexity of inference that predicts the peak energy-efficiency input/output length configuration. The model makes explicit how quadratic input costs and linear decoding costs combine to produce observed efficiency patterns, and it is extended beyond a FLOPs-only version to include memory-traffic effects.

2. Empirical validation across model families. Measurements using TensorRT-LLM on NVIDIA H100 GPUs across models from 1B to 9B parameters (OPT, LLaMA, Gemma, Falcon, Qwen2, Granite), with input and output lengths from 64 to 4096 tokens, achieving a mean MAPE of 1.79% (± 0.61).

3. Demonstration that efficiency is non-monotonic. Efficiency does not scale monotonically with sequence length; it exhibits distinct regimes, with peak efficiency at short-to-medium inputs combined with medium outputs, and sharp decline for long inputs or very short outputs. Sweet-spot configurations exceed the worst-case setting by a factor of 33.41×.

4. A closed-form predictor for the optimal output length. Setting the derivative of per-token energy with respect to output length to zero yields n_out* = sqrt((θ₁n_in² + θ₃n_in + θ₅) / θ₄), linking the optimal number of generated tokens directly to the input length through the fitted coefficients.

Main Findings

• Per-token energy is not constant: The paper contrasts its model against four baselines, including constant E_tok (Niu et al., 2025), an E_tok depending only on n_out, one depending on n_in + n_out (Wilhelm et al., 2025), and the linear workload model of Wilkins et al. (2024) — arguing none capture the non-linear interaction between input and output lengths.

• A generation energy minimum exists: The autoregressive structure creates a sweet spot where short-to-moderate inputs combine with medium-length outputs. Efficiency drops sharply for long inputs or very short outputs.

• Peak vs. worst case: Sweet-spot configurations achieve up to 33.41× better energy efficiency than the worst-case setting.

• Model accuracy: Mean MAPE of 1.79% (± 0.61) across the tested models and sequence-length grid (64 to 4096 tokens; 10, 100, and 1000 requests per workload).

• Inference dominates life-cycle energy: The paper cites prior work estimating inference can account for up to 90% of a model's life-cycle energy consumption.

• Memory, not just arithmetic, matters: The FLOPs-only formulation is extended with a memory-access decomposition; the paper notes attention kernels are often memory-bound, and the memory model introduces an additional constant term tied to the 10d² component of prefill memory traffic, which normalizes into a 1/n_out term.

• Framework choice affects the measurement baseline: Prior component-level measurements cited in the paper show vLLM and TensorRT-LLM consume about 5% of the GPU energy of a general-purpose serving framework such as Transformers, motivating TensorRT-LLM as the runtime for this study.

Methodology in Plain English

The researchers started from the Transformer architecture itself rather than from measurement curves. They broke inference into its two phases: prefill, where the whole input sequence is processed in parallel, and decode, where tokens are generated one at a time using cached keys and values. For each phase they wrote out the floating-point operations and the memory movements for the attention and feed-forward components of a single layer, then summed over all layers and over all generated tokens.

The result is an expression for total energy as a linear combination of polynomial terms in the input length and output length. Dividing by the number of output tokens converts this into energy per token, and differentiating that expression with respect to output length gives the formula for the most efficient number of tokens to generate.

To test the model, they built synthetic request datasets using the TensorRT-LLM utility script, sampling token sequences with the Llama-3.1-8B-Instruct tokenizer at fixed input and output lengths (zero variance) so the workload geometry was exactly controlled. They covered all combinations of input and output lengths from 64 to 4096 tokens in powers of two, at workload sizes of 10, 100, and 1000 requests. GPU power was sampled at 500 ms intervals through an updated ExaMon monitoring framework backed by IoTdb, using the NVIDIA Management Library Python API (pynvml) for metrics such as power usage and graphics clock. Total energy was computed by trapezoidal integration of instantaneous power over the measurement interval, then normalized to energy per token and its reciprocal, tokens per joule.

Why This Matters

Impact on research. The paper challenges the widely used assumption that LLM inference energy is linear in token count — the assumption behind per-token pricing and many existing energy models. By grounding the model in architectural complexity rather than curve fitting, it provides a principled framework that other researchers can extend to different attention mechanisms, model sizes, or hardware. It also argues that coarse-grained metrics like latency and throughput are insufficient for fine-grained inference energy profiling.

Real-world applications:

• Input truncation: Because long inputs push inference away from the sweet spot, production systems can truncate overly long prompts to recover efficiency.
• Summarization and adaptive generation: Aligning output lengths with the predicted peak configuration supports summarization and adaptive generation strategies that target the efficient regime.
• Batching strategy design: The model informs how deployments should tailor batching to workload geometry, complementing TensorRT-LLM's in-flight batching controls (max_batch_size and max_num_tokens).
• Cost and sustainability planning: The sweet-spot predictor lets operators estimate, before running workloads, whether a given request shape will be energy-efficient or wasteful.

Industry relevance. Commercial AI providers currently charge for inference based on generated output tokens or a linear combination of input and output tokens — a pricing structure this paper's findings suggest does not reflect the actual energy cost. Datacenter operators managing continuous inference workloads, where energy dominates both cost and environmental footprint, can use the model to schedule and shape requests rather than treating all tokens as equally expensive.

Future Directions

• Extending beyond the tested setup: The paper states that future work will extend the analysis to different inference frameworks beyond TensorRT-LLM, since it selected TensorRT-LLM to obtain a production-grade figure of merit.
• Broader hardware coverage: The current study is limited to NVIDIA H100 GPUs; whether the model's coefficients and sweet-spot location transfer to other accelerators is not addressed in the provided content.
• Scaling beyond 9B parameters: The test set spans 1B to 9B parameters across XS, S, and M families; how the model behaves for larger models is an open question.
• Translating the model into scheduling policy: The paper suggests production systems can exploit sweet spots through truncation, batching, and adaptive generation, but does not report a deployed system that does so.

Target Audience

Researchers and engineers working on LLM serving and inference efficiency; hardware and systems researchers interested in energy modeling for accelerators; datacenter and MLOps practitioners responsible for the cost and carbon footprint of inference deployments; and performance modelers who want an architecture-derived alternative to purely empirical energy curves. Readers need comfort with complexity analysis, matrix operation counting, and GPU power measurement to follow the derivations fully, though the conclusions about sweet spots and non-monotonic efficiency are accessible without them.

Authors’ abstract

Large Language Models (LLMs) inference is central to modern AI applications, dominating worldwide datacenter workloads, making it critical to predict its energy footprint. Existing approaches estimate energy consumption as a simple linear function of input and output sequence. However, by analyzing the autoregressive structure of Transformers, which implies a fundamentally non-linear relationship between input and output sequence lengths and energy consumption, we demonstrate the existence of a generation energy minima. Peak efficiency occurs with short-to-moderate inputs and medium-length outputs, while efficiency drops sharply for long inputs or very short outputs. Consequently, we propose SweetSpot, an analytical model derived from the computational and memory-access complexity of the Transformer architecture, which accurately characterizes the efficiency curve as a function of input and output lengths. To assess accuracy, we measure energy consumption using TensorRT-LLM on NVIDIA H100 GPUs across a diverse set of LLMs ranging from 1B to 9B parameters, including OPT, LLaMA, Gemma, Falcon, Qwen2, and Granite. We test input and output lengths from 64 to 4096 tokens and achieve a mean MAPE of 1.79%. Our results show that aligning sequence lengths with these efficiency "sweet spots" reduce energy usage, up to 33.41x, enabling informed truncation, summarization, and adaptive generation strategies in production systems.