Deadline-Aware, Energy-Efficient Control of Domestic Immersion Hot Water Heater

AI summary

Overview

Research area: Reinforcement learning and model-based planning applied to domestic hot-water heating control (building energy management, deadline-aware control).

Technical level: Intermediate. The physics model is deliberately simple (first-order thermal losses), but the paper assumes familiarity with Markov decision processes, Monte Carlo Tree Search, and Proximal Policy Optimisation.

Scope: The paper builds a small Gymnasium benchmark for controlling a domestic immersion hot-water heater and compares three controllers — a time-optimal bang-bang baseline, a zero-shot MCTS planner, and a trained PPO policy — on energy used to reach a target temperature by a specified deadline under identical physics and timing.

What This Paper Is About

Domestic immersion water heaters are often simply left on during winter, heating quickly rather than efficiently, ignoring predictable demand windows and the heat that leaks away to the room in the meantime. The paper's goal is to test whether a controller can reach a specified target temperature at a specified time while expending as little energy as possible, using only discrete on/off actions at a fixed 6000 W power level applied every 120 s. It frames this as a finite-horizon decision problem and asks which style of controller — a simple rule, an online planner, or a learned policy — does best when the physics are held constant.

Key Contributions

1. A deadline-aware, energy-minimising immersion-heater benchmark: a Gymnasium environment with first-order thermal losses, discrete on/off actuation at 6000 W applied every 120 s, and a transparent evaluation protocol in which physics, initial conditions, and timing are identical across controllers.
2. A head-to-head comparison of three controllers — a time-optimal bang-bang baseline, a zero-shot Monte Carlo Tree Search planner, and a Proximal Policy Optimisation policy — on the same environment, reporting total energy in Wh.
3. Evidence from one-dimensional sweeps over target temperature (40–80 °C), initial temperature (10–30 °C), and deadline (30–90 steps) showing that the learned policy forms the lower energy envelope while model-based planning provides only partial savings over the baseline.
4. A discussion of deployment trade-offs: MCTS offers training-free improvement but requires online search at every step, whereas a trained policy executes instantly and is easier to embed at scale.

Main Findings

• PPO is the most energy-efficient controller. At a 60-step horizon (2 h), PPO uses 3.23 kWh, against bang-bang's 4.37–10.45 kWh and MCTS's 4.18–6.46 kWh, yielding savings of 26% at 30 steps and 69% at 90 steps.
• Savings grow as the deadline lengthens. PPO stays near-flat at approximately 3230 Wh across horizons, while bang-bang increases roughly linearly from 4.37 to 10.45 kWh (about 1520 Wh per +15 steps, ≈101 Wh/step). MCTS is intermediate at about 4.18–6.46 kWh with non-monotonic increments.
• The representative 60-step, 60 °C trajectory shows the same ordering. With m = 50 kg and T_a = 20 °C, PPO consumes 54% less energy than bang-bang and 33% less than MCTS.
• Mechanism behind the gap. PPO follows a delayed heating strategy, deferring heating until necessary and concentrating control effort near termination; bang-bang heats at full power from the start and then oscillates within the target band, losing energy while maintaining temperature; MCTS applies targeted bursts but can overshoot or mis-time terminal adjustments because of randomness in its rollouts.
• Sensitivity to starting conditions. Warmer start states reduce energy for every controller, but PPO shows weak sensitivity and minimal run-to-run dispersion, MCTS is intermediate with higher dispersion and some non-monotonicity, and bang-bang remains highest throughout.
• Target temperature sensitivity. Energy rises for all methods as the target rises from 40 to 80 °C; PPO is consistently lowest, MCTS intermediate, and bang-bang highest with the steepest growth.
• Terminal accuracy. Bang-bang achieves acceptable terminal temperatures but at systematically higher energy; MCTS shows higher terminal-temperature variance and occasional overshoot or undershoot; PPO occupies the Pareto-efficient frontier.
• Reporting caveat. The authors do not filter runs by success — energy is reported for all runs, and a representative trajectory that reaches the target at the deadline is shown.

Methodology in Plain English

The authors simulate a single water tank with lumped thermal capacity, assuming the water temperature is spatially uniform. Heat enters from an electric immersion element with fixed efficiency, and heat leaves by convection to the surrounding air, giving a first-order energy balance. Forward-Euler integration at a 120 s time step yields a simple update rule for the next temperature. With the nominal parameters, the dimensionless cooling factor is about 0.043, which the authors note is well inside the stable range for explicit integration.

The environment is fully observable, and the observation given to every controller is four scalars: current temperature, target temperature, ambient temperature, and the number of steps remaining until the deadline. The action space is discrete — off or on, mapped to 0 or 6000 W. Episodes terminate at the target time step within a tolerance of at least one step, and the service constraint checked at the deadline is |T_D − T_target| ≤ ±1 °C.

The reward is per-step plus terminal. Each step costs a small penalty proportional to energy used, with α = 1.86 × 10⁻⁸ J⁻¹; at the end, a penalty proportional to the temperature error is applied with β = 0.03 per °C. With P_on = 0.95 × 6000 = 5700 W and a 120 s step, one on-step costs 684,000 J, so α times step energy is about 0.0128 — deliberately smaller than the 0.03 benefit of reducing terminal error by 1 °C, so that heating at the penultimate step is worthwhile when it actually helps. The authors note that if the expected improvement is small (below 0.4 °C), the action is not justified, discouraging gratuitous heating.

Three controllers are compared. The bang-bang baseline applies full power until the temperature enters the target band and then holds within the threshold; it is time-optimal but not energy-optimal. MCTS plans over the evaluation horizon using the four canonical phases (selection with UCB1, expansion, simulation, backup), with exploration constant c = √2 ≈ 1.414 and 25,000 simulations per episode; the authors report that budgets below this increased variance and improvement beyond it was marginal. PPO comes from Stable-Baselines3 with a multilayer perceptron and a discrete action head, trained under default hyperparameters for 2.5 million environment steps, with the authors reporting convergence at around 2.1 million steps; each episode drew a fresh start state for generalisation. All implementations are in Python, with training and evaluation on CPU.

Evaluation holds physics, initial conditions, and timing constant, reports total energy in Wh, and runs three one-dimensional sweeps: target temperature over {40, 50, 60, 70, 80} °C, initial temperature over {10, 15, 20, 25, 30} °C, and deadline over {30, 45, 60, 75, 90} steps. A representative trajectory uses T_0 = 20 °C, T_target = 60 °C, and D = 60 steps.

Why This Matters

The work isolates a narrow but practical question — arrive on time, use as little energy as possible — and shows that anticipation, not raw power, drives savings. Because the physics are identical across controllers and the evaluation protocol is fixed and tolerance-based, the comparison is method-agnostic and reusable.

Real-world applications:

• Domestic immersion hot-water tanks with simple on/off relay hardware, where a trained policy could be embedded at low inference cost.
• Peak-load shifting for utilities and aggregators, since a controller that heats just in time rather than continuously reduces prolonged full-power operation.
• Optimal-start style heating in building controls, where the goal is to reach a temperature band at a specified occupancy time.
• Benchmarking of control algorithms for thermostatically controlled loads, using the Gymnasium environment as a small, reproducible testbed.

Industry relevance: The planner-versus-policy trade-off is framed as practical rather than philosophical. MCTS requires online search at every decision step, which the paper notes can exceed embedded-controller budgets; a trained PPO policy executes instantly once trained. The paper is explicit about scope limits: there is no time-of-use pricing, no emissions signal, no stratified tank model, and no modelling of user behaviour or draw events, so the reported metric is energy alone, with cost and emissions left as extensions.

Future Directions

• Extending to continuing on-demand control with periodic truncation rather than fixed episodic deadlines.
• Incorporating time-varying tariffs to assess cost alongside energy.
• Adding richer actuation and assessing cost, emissions, and peak power, not just total energy.
• Open questions raised by the scope limits: how results change with stratified tank models, draw events, or other disturbances, and whether the learned policy's advantage persists under richer dynamics that the paper deliberately excluded.

Target Audience

Researchers and practitioners in building energy management, reinforcement learning for control, and smart-home or demand-side management who want a compact, reproducible comparison of rule-based, planning, and learned controllers. It is also useful for readers with some RL background who want a concrete example of reward shaping for a deadline-constrained physical task, and for engineers evaluating whether to deploy online search or a trained policy on low-cost heater hardware.

Authors’ abstract

Typical domestic immersion water heater systems are often operated continuously during winter, heating quickly rather than efficiently and ignoring predictable demand windows and ambient losses. We study deadline-aware control, where the aim is to reach a target temperature at a specified time while minimising energy consumption. We introduce an efficient Gymnasium environment that models an immersion hot water heater with first-order thermal losses and discrete on and off actions of 0 W and 6000 W applied every 120 seconds. Methods include a time-optimal bang-bang baseline, a zero-shot Monte Carlo Tree Search planner, and a Proximal Policy Optimisation policy. We report total energy consumption in watt-hours under identical physical dynamics. Across sweeps of initial temperature from 10 to 30 degrees Celsius, deadline from 30 to 90 steps, and target temperature from 40 to 80 degrees Celsius, PPO achieves the most energy-efficient performance at a 60-step horizon of 2 hours, using 3.23 kilowatt-hours, compared to 4.37 to 10.45 kilowatt-hours for bang-bang control and 4.18 to 6.46 kilowatt-hours for MCTS. This corresponds to energy savings of 26 percent at 30 steps and 69 percent at 90 steps. In a representative trajectory with a 50 kg water mass, 20 degrees Celsius ambient temperature, and a 60 degrees Celsius target, PPO consumes 54 percent less energy than bang-bang control and 33 percent less than MCTS. These results show that learned deadline-aware control reduces energy consumption under identical physical assumptions, while planners provide partial savings without training and learned policies offer near-zero inference cost once trained.