The Environmental Impacts of Language Model Training Keep Rising Now is the Time to Catch Impacts on the Rebound

AI summary

Overview

Research area: Environmental impact assessment of machine learning and natural language processing, specifically life-cycle assessment (LCA) of the hardware and energy used to train large language models.

Technical level: Intermediate. The paper assumes familiarity with concepts such as FLOPs, GPU-hours, PUE, and carbon intensity, but explains its LCA methodology and impact indicators (GWP, ADPe) in self-contained terms.

One-sentence scope: The paper estimates the carbon footprint and metallic resource depletion of training 234 ML models documented in the Epoch AI database over roughly a decade, and tests whether hardware, software, and carbon optimizations have reduced those impacts.

What This Paper Is About

Prior work reported the carbon cost of individual training runs, but no comprehensive analysis existed of how training impacts have trended across many models over time. The authors combine a life-cycle assessment tool (MLCA) with the Epoch AI Notable systems database and GPU manufacturer data to reconstruct training duration, energy use, carbon footprint, and metallic resource depletion for models released between roughly 2013 and 2025. They then ask whether optimization strategies—more energy-efficient hardware, more efficient algorithms, and lower-carbon electricity—have actually lowered the impacts of training language models.

Key Contributions

1. A decade-scale impact trend analysis. The authors assemble training impact estimates for 234 models (26% of the Epoch AI database), including 132 NLP models, of which 123 are language models, and use regression analysis to characterize trends in energy use, GWP, and ADPe over time.
2. Two methods for estimating missing training durations. They define GPU-h₁ (training duration × number of cards) and GPU-h₂ (training FLOP divided by peak card compute power times 0.27), validate them against each other, and use a fitted linear model to correct the systematic underestimation of GPU-h₂.
3. A decomposition of impacts across hardware life-cycle phases. They quantify the share of embodied (production-phase) impacts in total impacts, showing that metallic resource depletion comes almost exclusively from hardware production, and that embodied GWP is typically about a quarter to a third of the total.
4. An evaluation of three optimization strategies. They test hardware efficiency, software efficiency, and simulated carbon-intensity reductions of up to 25% per year from 2019, and argue the observed patterns are consistent with a rebound effect. They also propose a standard reporting table for the information needed to reproduce environmental assessments.

Main Findings

• Impacts rose exponentially. Estimated energy use, carbon footprint (GWP), and metallic resource depletion (ADPe) of training ML models all increased exponentially over the studied period, as shown in the logarithmic-scale trend figures.
• NLP models are representative. Additional disaggregated experiments indicate language, vision, and multimodal models (to a lesser extent) all exhibit exponential growth trends.
• Embodied impacts are large and unequal across indicators. The mean share of embodied ADPe on total ADPe is 99%, ranging from a minimum of 89% to a maximum of 100%. The mean share of embodied GWP is 23%, ranging from 12% to 58%. ADPe comes close to exclusively from hardware production.
• Hardware efficiency improved but did not reduce energy use. Compute efficiency of the graphics cards used (FLOPs per Watt) increased exponentially between 2013 and 2025, yet the use of more compute-efficient hardware is correlated with an exponential increase in hardware energy use during training.
• Software efficiency gains were absorbed. Software efficiency (FLOP per parameter) showed a stable or even increasing trend over time, when a decrease would be expected from more efficient newer architectures, indicating that algorithmic gains served model growth rather than impact mitigation.
• Combined efficiency stayed nearly flat. The combined efficiency of training, expressed in kWh per parameter, appears mostly constant with only a slight improvement with model size.
• Carbon optimization is insufficient. Comparing actual electricity mixes with a simulated 25% annual reduction in carbon intensity shows the carbon footprint of models released from 2019 increasing regardless of the scenario, with significantly positive regression coefficients in both cases.
• Carbon intensity has a floor. Electricity carbon intensity is limited by the lowest global mixes at 15–20 gCO₂eq/kWh, so even fast reductions cannot keep pace with rising energy use.
• Hardware count grew exponentially. The number of graphics cards used for training increased exponentially over time, and cards themselves have higher production impacts as they become more technologically advanced.
• Worked example, GPT-4. Training was estimated at 57,000,000 GPU-hours on NVIDIA A100 SXM4 40 GB cards using GPU-h₁. Total estimated impacts: 32.8 GWh energy, 15 ktCO₂eq GWP, and 300 kgSb eq ADPe, split into embodied (3.3 ktCO₂eq, 300 kgSb eq), usage (27.4 GWh, 10.2 ktCO₂eq, 2.7 kgSb eq), and infrastructure (5.4 GWh, 2.0 ktCO₂eq, 0.5 kgSb eq). Embodied impacts represent 20% of the total carbon footprint and close to 100% of metallic resource depletion for this model.
• Estimated trends are statistically supported. Regression analysis (detailed in the appendices) backs the assessed trends; the correction model for training duration had an adjusted R² of 0.98 and a fitted performance ratio of approximately 27%.

Methodology in Plain English

The authors needed two things for each model: a description of the hardware used and how long it ran. They took hardware and training information from the Epoch AI Notable systems database, which they describe as the most comprehensive database on ML systems, and graphics card characteristics from a dataset gathering manufacturer information.

To get training duration, they used whichever of two approaches was available. The first multiplies reported training duration by the number of cards, giving GPU-h₁; this is the most reliable because it comes from the papers presenting the models, but it covers only 15% of Epoch AI models. The second divides training FLOP by the card's peak compute power, then corrects for the fact that hardware does not run at peak continuously. The authors compared the two on the 119 models where both were available, excluded 19 anomalies (16%) where the ratio exceeded one or fell below 10%, and fit a linear model on 100 observations to derive the correction, which corresponds to a quasi-constant performance ratio of about 27%. The final GPU-hour estimate uses GPU-h₁ for 131 models (15%) and GPU-h₂ for another 103 models (11%), covering 26% of Epoch AI models.

Impacts were then computed with the MLCA tool, which uses attributional life-cycle assessment and models hardware bottom-up as a sum of parts. Graphics cards are modeled from GPU size, on-board memory size, and a constant value for other components. Energy use during training is estimated from the card's Thermal Design Power and the training duration. Carbon footprint is expressed as Global Warming Potential over a 100-year horizon in kgCO₂eq; metallic resource depletion is expressed as Abiotic Depletion Potential for elements in kilograms antimony equivalent. The assessment covers server production, server usage, and data center cooling. Server configurations were assumed (for example, 4 graphics cards, 2 CPUs and 512 GB memory for servers with NVIDIA workstation cards; 2 cards, 2 CPUs and 192 GB memory otherwise), a hardware lifespan of 3 years was used, GPU and CPU usage was set at 100%, and PUE was interpolated for 2010–2018 with a value of 1.2 from 2018 onward. Electricity impacts were based on the energy mix of the countries of the model producers documented by Boavizta, with intervals built when multiple countries were involved. Eight models were excluded because hardware was used inconsistently across training.

For the optimization analysis, the authors computed hardware compute efficiency as peak performance divided by TDP, software efficiency as average compute per parameter, and combined efficiency in kWh per parameter. For carbon optimization, they simulated continuous reductions in the carbon intensity of electricity starting from 2019 for models released from that year onward.

Why This Matters

The paper argues that efficiency improvements alone do not deliver sustainability for model training, and that the field currently lacks the information needed to reproduce environmental assessments. Reporting only a carbon number is described as insufficient; the authors call for environmental impact to be integrated into NLP evaluation and into research planning and decision making.

Real-world applications:

• Model release documentation: the proposed standard reporting table (training duration, card model, number of cards, server model and configuration, datacenter type, hardware usage ratio, datacenter PUE, datacenter location) gives labs a concrete checklist for what to publish alongside a model.
• Datacenter planning and hardware procurement: the finding that ADPe is almost entirely embodied, and that frequent hardware replacement shifts impacts rather than removing them, is directly relevant to refresh cycles and to life-cycle cost accounting for compute facilities.
• Green AI strategy evaluation: organizations weighing renewable procurement or geographic relocation of training can use the carbon-intensity simulation results to see the limits of those levers against exponentially growing energy demand.
• Benchmark and leaderboard design: the argument that NLP evaluation over-emphasizes task performance provides a rationale for adding environmental impact as a reported dimension.

Industry relevance is direct: the major AI companies cited in the paper have claimed that hardware, algorithmic, and carbon optimizations mitigate or reduce the carbon footprint of ML, and this study tests those claims against a decade of training data.

Future Directions

• Extending assessments to inference. The authors note that inference weighs more heavily over the life cycle of models and that earlier work by de Vries and by Desislavov et al. suggests their findings should also apply there, but that assessing inference requires access to user information.
• Accounting for model development. Model development can involve training multiple smaller models before converging on a final architecture; the authors cite Morrison et al. (2025) in stating these multiple training runs, which they do not account for, can account for at least half of the total footprint.
• Broadening the impact indicators. Water usage, toxicity to human and non-human life, ecosystem destruction, and pollution from hardware mining and disposal are named as important impacts excluded here, in part because recent open quality information on them and on end-of-life is missing.
• Standardized, reproducible reporting. The paper's proposed reporting format is posed as a starting point; adopting it community-wide, and thereby making environmental assessment reproducible at scale, remains an open task.

Target Audience

Researchers in NLP and machine learning who design, train, or evaluate large models and want to understand the environmental consequences of those choices; sustainability and green AI researchers and practitioners who need trend data rather than single-model case studies; policy analysts and datacenter or infrastructure planners evaluating efficiency claims; and reviewers or venue organizers considering whether environmental impact should be part of standard evaluation and reporting practice.

Authors’ abstract

Recent Machine Learning (ML) approaches have shown increased performance on benchmarks at the cost of escalating compute demands. Hardware, algorithmic and carbon optimizations have been proposed to curb energy use and environmental impacts. We estimate the environmental impacts associated with training models documented in the Epoch AI database over the last decade, with a particular focus on impacts associated with Large Language Models and the hardware used to train them. We find that energy use and environmental impacts associated with training ML models have increased exponentially, even when considering impact reduction strategies such as using less carbon intensive electricity mixes or more efficient hardware. Optimization strategies do not mitigate the impacts induced by model training, suggesting rebound effect. We show that the impacts of hardware must be considered over the entire life cycle rather than the sole use phase in order to avoid impact shifting. Our study demonstrates that increasing efficiency alone does not ensure sustainability. There is an urgent need to systematically integrate environmental impacts in NLP evaluation practices to better inform the community and support the use of impact as a feature in research planning and decision making.