Natural-gas storage modelling by deep reinforcement learning

AI summary

Overview

• Research area: Machine learning applied to economics — specifically deep reinforcement learning for energy-market simulation, natural-gas storage operations, and regulatory policy analysis.
• Technical level: Intermediate. The paper combines reinforcement learning (SAC, PPO, DDPG, TD3, A2C) with a reduced-form economic market model; readers need basic familiarity with RL concepts and market microeconomics, but the presentation is largely conceptual and self-contained.
• Scope: The paper introduces GasRL, a simulator that pairs a calibrated stochastic model of the Italian natural-gas market with a deep-RL storage-operator agent, and uses it to study how optimal stockpile management shapes prices, demand, supply, and the effects of EU-mandated storage thresholds.

What This Paper Is About

European regulators require underground natural-gas storage to be refilled to high levels before winter (at least 90% by 1 November each year, with 80% for the transitional year 2022, under the June 2022 EU rule; the July 2025 revision allows the target to be met anytime between 1 October and 1 December). This obligation is contested, because a rigid target may distort seasonal prices, inflate summer wholesale prices, and burden countries with large storage capacity. The paper builds a simulator in which a single monopolistic storage operator (a realistic assumption for Italy, where SNAM owns around 90% of national storage capacity) learns a pricing-and-inventory policy by deep reinforcement learning, and uses it to trace how storage behaviour feeds back into prices, profitability, and market resilience.

Key Contributions

1. The GasRL simulator. A two-part tool coupling (a) a demand-and-supply environment calibrated to the Italian gas market with (b) a deep-RL storage operator that sets prices and bridges demand–supply imbalances through inventory, subject to capacity limits.
2. A multi-objective reward design for a storage operator. The reward encodes profitability, price-volatility minimisation, near-absolute market clearing, and compliance with a government-mandated minimum storage threshold — reflecting the operator's dual private–public character.
3. A benchmark of five RL algorithms on the environment. DDPG, TD3, A2C, SAC, and PPO are compared, with SAC identified as the best performer in terms of reward level and learning stability.
4. A policy experiment on storage thresholds. The simulator is used to compare a baseline model (θ_n = 0) with a regulated model (θ_n = 1000) enforcing an 83% November refilling threshold, measuring market success, profitability, price volatility, and price level under varying supply-shock volatility.

Main Findings

• SAC is the strongest algorithm for GasRL. SAC, DDPG, and PPO reach very similar rewards after 1.5 million training steps, but SAC outperforms the others in learning stability. TD3 and PPO underperform SAC on stability, with TD3 showing much larger reward fluctuations around the mean. DDPG challenges SAC up to roughly 50k steps before deteriorating. A2C, possibly due to high hyperparameter sensitivity, is found incapable of learning under the standard parameterisation.
• A fully trained policy behaves very differently from a poorly trained one. At 4000 training steps the agent already sets prices according to the season, showing large periodic oscillations, but this cyclical strategy is unprofitable (the bank account slopes downward). At 1.5 million steps the price policy is more nuanced, less seasonally driven, more responsive to current market shocks, and the bank account slopes upward.
• Performance metrics converge with training. The mean episodic reward at the end of the 30-year test horizon rises steadily with training steps; the bank account improves substantially but unevenly; price volatility falls steadily; and market failures are virtually eliminated over the 30-year horizon. The market-success metric levels off at around 32,000 learning steps.
• Stockpiles are reasonable even without regulation. With no compliance enforcement (θ_n = 0), November inventories reach the 83% threshold early in training, then settle around 2.2 (73% of storage capacity), after rising to 2.5 (83% of capacity) at the beginning of training.
• Simulated prices match real-world volatility and seasonality without price calibration. Against TTF front-month gas futures prices recorded at the end of each month over 2010–2024, the simulator reproduces the seasonal pattern (peak in November for both real and simulated series, coinciding with the refilling deadline), although seasonal troughs occur slightly earlier in simulations. The standard deviation of the simulated log-price difference is 27%, versus 17% in the full 2010–2024 historical sample and 25% in the 2020–2024 period. No price series was used in training.
• An 83% threshold improves resilience at a cost. Comparing baseline (θ_n = 0) and regulated (θ_n = 1000) models trained at σ_s = 0.04 and tested up to σ_s = 0.07 (75% above the training value, with 1000 repetitions), the regulated model shows a small but statistically significant improvement in market success (1 − m_t), larger at higher supply-shock volatilities. The costs are lower profitability (a lower bank account) and slightly higher price volatility, significant at low supply-shock volatilities. The regulatory constraint has no measurable effect on the average price level.

Methodology in Plain English

The authors build the simulator in two stages. First, they create a monthly market environment for Italian gas. A price set by the agent is combined with past price signals through an exponentially weighted moving average, producing "sticky" demand and supply responses, because agents adjust behaviour gradually. Demand depends on a seasonal component estimated from Italian monthly gas consumption (a truncated Fourier series) minus an elasticity term, plus a persistent AR(1) shock; supply depends on an elasticity term plus its own AR(1) shock. Excess demand is the difference between supplied and demanded quantities, and the storage operator absorbs it: inventories rise on injection and fall on withdrawal, bounded by maximum capacity. If demand exceeds available inventory, or if desired accumulation exceeds capacity, a market failure is recorded.

Second, they model the storage operator as a reinforcement-learning agent. Its action is a single continuous scalar, exponentiated to give the price; its nine-component state includes seasonality signals, the month of year, demand and supply shocks, demand and supply price signals, the log of the inventory, and the current action. The reward adds the change in a bank account (which earns interest, pays a proportional storage cost, and settles trades at the chosen price) and subtracts penalties for price volatility, market failures (with severity-weighted terms), and failure to meet a November inventory threshold. Gas left at the end is liquidated at the midpoint of demand and supply log-price signals.

The environment was implemented with the Gymnasium interface and agents trained with Stable-Baselines3. For each of the five algorithms, the paper reports five independent training runs of 1.5 million steps with default hyperparameters, and separately describes a comparison protocol of 10 training runs with different seeds per model, 5 test runs per seed, and mean-with-standard-error reporting across seeds. Robustness checks covered learning rates, checkpointing and selection strategies, and actor–critic network sizes (up to 3 hidden layers, and 2x or 4x more neurons), with no significant performance change. All reported results use the SAC agent trained until convergence, with 50 test runs for average trajectories, and the policy-relevant comparisons with 1000 repetitions.

Why This Matters

• Research impact. The paper shows that an RL agent trained only on a calibrated market environment can endogenously generate price dynamics — volatility and seasonality — that resemble real gas prices, without being trained on any price series. This supports the use of RL as a tool for modelling strategic economic agents in energy markets, complementing existing approaches such as partial-equilibrium models, PLEXOS-based scenario studies, and deep-learning storage optimisation.
• Real-world applications.
  • Assessing the market effects of EU storage-refilling mandates, including the debated 90% November target and proposals for a looser 83% threshold.
  • Stress-testing market resilience under supply shocks more severe than those seen in the calibration sample.
  • Studying the trade-off between operator profitability, price stability, and the public-service role of storage.
  • Providing regulators and market participants with a sandbox for evaluating regulatory design before implementation.
• Industry relevance. Large storage operators such as SNAM, which owns around 90% of Italian storage capacity, are the modelled decision-makers. The framework speaks directly to the profitability-versus-stability trade-off these operators face, and to how their injection and withdrawal timing shifts the net balance of supply and demand.

Future Directions

• GPU-accelerated training using frameworks such as rlax or Jax, to allow more extensive hyperparameter tuning and finer calibration of the environment to real-world data.
• More in-depth exploration of policy implications across additional regulatory scenarios, beyond the baseline versus θ_n = 1000 comparison presented here.
• Robustness testing of learned policies under alternative environment specifications, including different demand and supply shock processes, different elasticity estimates, and alternative institutional constraints.
• Multi-agent extensions, adding competing storage operators or traders, and explicitly modelling the international dimension of the gas market, since the current version treats Italy as a closed system in which cross-border interaction enters only through the reduced-form demand and supply equations.

Target Audience

Economists and central-bank or regulatory analysts working on energy markets and gas-storage regulation; machine-learning researchers interested in RL for economic agent modelling and in applied RL benchmarks; energy-market practitioners and policymakers evaluating storage obligations; and graduate students at the intersection of reinforcement learning and applied microeconomics who want a concrete, reproducible case study (code is available in Python and in a Julia reimplementation).

Authors’ abstract

We introduce GasRL, a simulator that couples a calibrated representation of the natural gas market with a model of storage-operator policies trained with deep reinforcement learning (RL). We use it to analyse how optimal stockpile management affects equilibrium prices and the dynamics of demand and supply. We test various RL algorithms and find that Soft Actor Critic (SAC) exhibits superior performance in the GasRL environment: multiple objectives of storage operators - including profitability, robust market clearing and price stabilisation - are successfully achieved. Moreover, the equilibrium price dynamics induced by SAC-derived optimal policies have characteristics, such as volatility and seasonality, that closely match those of real-world prices. Remarkably, this adherence to the historical distribution of prices is obtained without explicitly calibrating the model to price data. We show how the simulator can be used to assess the effects of EU-mandated minimum storage thresholds. We find that such thresholds have a positive effect on market resilience against unanticipated shifts in the distribution of supply shocks. For example, with unusually large shocks, market disruptions are averted more often if a threshold is in place.