Reinforcement Learning and Rule-Based Peer-to-Peer Pricing in Residential PV-BES Communities

AI summary

Overview

Research area: Peer-to-peer (P2P) electricity trading in residential energy communities, comparing rule-based pricing mechanisms against reinforcement-learning (RL) pricing, in communities with rooftop photovoltaic (PV) generation and, optionally, battery energy storage (BES).

Technical level: Intermediate. The core comparison is readable without a specialist background, but the RL section assumes familiarity with Deep Q-Networks, while the market section assumes familiarity with local energy market settlement and grid tariff structures.

Scope in one sentence: The paper benchmarks three rule-based P2P pricing rules (bill-sharing, mid-market rate, and supply–demand-ratio pricing) against three DQN-based pricing modes across a 20-household Polish community over one annual hourly horizon, in a PV-only and a PV-BES configuration.

What This Paper Is About

Local electricity communities need a rule for setting the price at which neighbours trade electricity with each other, and the choice of rule determines who gains and who loses. Rule-based mechanisms such as the mid-market rate and supply–demand-ratio pricing are well established, while reinforcement learning has been proposed as a way to adapt prices automatically to changing conditions, but direct financial comparisons between the two families are limited. This paper runs that comparison head-to-head on the same community, the same demand data, and the same PV portfolio, and asks whether a learned pricing policy actually beats simple, transparent rules.

Key Contributions

1. A controlled head-to-head comparison of rule-based and RL-based P2P pricing under identical physical conditions. Three rule-based benchmarks (bill-sharing, mid-market rate, SDR) and three RL modes (multiplier-based RL-M, fixed-parameter RL-SDR-F, learnable RL-SDR-L) are evaluated on the same 20-household community, the same demand realization, and the same PV portfolio, with a shared community reference cost of €4,747.81.

2. A non-learning control inside the same pricing family. RL-SDR-F uses the SDR-shaped price mapping with a single fixed sensitivity pair (α, β) = (0.5, 2.0) and an action set of size K = 1, which isolates the value of adapting α to operating conditions from the value of the SDR-shaped mapping itself.

3. A settlement-design explanation for why RL underperforms the rule-based benchmarks. The paper attributes the gap to budget balance rather than to learning quality: bill-sharing, MMR, and SDR all settle at a single internal price so that the entire grid price spread stays within the community, while the RL modes quote separate buy and sell prices and retain an internal spread that leaves the participants.

4. Decomposition of storage benefits and of RL-M's trading shortfall. Storage savings are separated from settlement savings, and the volume RL-M fails to clear internally is split into a portion caused by the action set (1,768 kWh) and a portion caused by the learned policy selecting an inverted spread when a feasible option existed (1,429 kWh).

Main Findings

• Rule-based pricing wins in the PV-only base case. Bill-sharing, MMR, and SDR each achieve community savings of €829.98 (17.48% of the €4,747.81 reference cost). The best RL policy, RL-SDR-L, reaches €734.23 (15.46%), followed by RL-SDR-F at €616.26 (12.98%) and RL-M at €418.49 (8.81%).

• SDR-shaped pricing beats the multiplier parameterization in every comparison. Across both configurations and across learning-based modes, the SDR-shaped formulations outperform the multiplier-based one; RL-M is the weakest policy in both the PV-only case (€418.49) and the PV-BES case (€606.94).

• Storage substantially improves the learning-based results, evaluated for RL policies only. Savings rise from €418.49 to €606.94 for RL-M, from €616.26 to €860.09 for RL-SDR-F, and from €734.23 to €978.52 for RL-SDR-L (20.61%). With batteries but no internal trading, community cost falls to €4,642.05, so €105.76 of the PV-BES savings comes from storage alone; internal settlement accounts for the remaining €872.76 under RL-SDR-L, which is €138.53 more than the same policy achieves in PV-only.

• The RL advantage comes from settlement efficiency, not from trading more energy. RL-SDR-F and RL-SDR-L clear the same 7,131 kWh and reach the same self-sufficiency index of 0.301 in PV-only. The annual internal spread falls from €210.46 under RL-SDR-F to €92.49 under RL-SDR-L, a reduction of €117.97 that matches the community-savings gap exactly. In PV-BES, RL-SDR-L exceeds RL-SDR-F by €118.43 on the same traded volume. The learned policy selects a mean α of 0.26 in surplus intervals and 0.48 in deficit intervals.

• Rule-based mechanisms clear more energy and reach higher self-sufficiency in PV-only. BS, MMR, and SDR all clear 7,266 kWh with an SSI of 0.304 and 26,263 kWh of grid imports. RL-SDR-F and RL-SDR-L clear 7,131 kWh (SSI 0.301, 26,398 kWh imports), while RL-M clears only 4,069 kWh (SSI 0.219, 29,459 kWh imports). In PV-BES, RL-SDR-F and RL-SDR-L clear 7,644 kWh with SSI 0.325 and 25,459 kWh imports, versus RL-M at 4,624 kWh, SSI 0.245, and 28,479 kWh.

• The mid-market rate dominates RL-SDR-L on every reported financial metric in PV-only. MMR attains €95.75 more community savings, a lower user cost (€6,043.68 versus €6,095.92) and higher prosumer revenue (€2,125.85 versus €2,082.34). The cheapest average trading price in PV-only is SDR at 0.106 €/kWh, followed by RL-M at 0.120, RL-SDR-F at 0.140, MMR at 0.141, and RL-SDR-L at 0.147.

• Benefits are distributed very unevenly across households. On the same aggregate savings of €829.98, SDR splits the benefit most evenly, with consumer and prosumer median savings of 19.3% and 28.6%, whereas bill-sharing and MMR give consumer medians near 10% and prosumer medians above 100%, the latter resting on small net-bill denominators.

• The RL-M shortfall in PV-only has two distinct sources. Of the energy not settled internally under RL-M, 1,768 kWh falls in intervals where no pair in the action set yields a valid spread, and 1,429 kWh falls in intervals where at least one feasible pair existed but the learned policy chose an inverted one.

• The PV-BES comparison is restricted within the RL family. The rule-based benchmarks are not re-evaluated under storage, so the PV-BES numbers are not directly comparable with the PV-only rule-based values. Because the battery trajectory is fixed independently of the settlement mechanism, the paper states that the budget-balance bound indicates the rule-based benchmarks would retain their advantage under storage.

Methodology in Plain English

The authors build a simulated community of 20 households — 8 prosumers with rooftop PV and 12 consumers without generation — and run it hour by hour for a full year, giving 8,784 hourly intervals drawn from a 366-day dataset. Household demand profiles come from the RAMP bottom-up stochastic load model, calibrated to Polish households using national statistical office data, and hourly PV output comes from Renewables.ninja irradiance and temperature series for Poland, converted through a capacity-based model with temperature derating. Households pay 0.249 €/kWh in weekday peak zones (06:00–13:00 and 15:00–22:00) and 0.137 €/kWh otherwise, under a G12w time-of-use tariff, while exports are valued at the hourly market price used for Polish net-billing, ranging from zero to 0.657 €/kWh with an annual mean of 0.100 €/kWh. In 384 of the 8,784 intervals the export price exceeds the import price, inverting the price corridor.

For each hour, the amount traded internally is limited by the smaller of the community's aggregate surplus and its aggregate residual demand. The rule-based mechanisms settle that volume in full every interval, at a single internal price bounded by the grid import and export prices. The RL mechanisms instead post separate buy and sell prices in advance, and only trade in intervals where those prices sit strictly inside the grid price corridor with a non-negative spread; otherwise everyone settles with the grid.

The RL agent is a Deep Q-Network that observes a compact community state — encoded hour of day, a discretized net balance with nine bins, a five-bin discretization of mean battery state of charge, and a bounded logarithmic transform of the lagged supply–demand ratio — and picks one of K pricing actions. In RL-M, actions are buy and sell multipliers applied to grid prices (K = 8). In RL-SDR-L, actions are sensitivity pairs (α, β) in an SDR-shaped price rule (K = 9). RL-SDR-F uses a single fixed pair (K = 1), so it cannot learn. The reward is the hourly cost saving relative to a no-P2P reference, which makes the agent maximize aggregate community savings. The network uses two hidden layers of 128 and 64 ReLU units, Adam with a learning rate of 5×10⁻⁴, minibatches of 64 drawn from a replay buffer of 50,000, a discount factor of 0.95, and target-network synchronization every 200 steps. Training runs for 500 episodes, each on one randomly sampled day, with ε decaying from 1.0 to 0.01 at a rate of 0.99952 per step. Policies are then evaluated greedily over the same annual dataset. The reported results come from a single training run per mode and configuration with a fixed random seed, so differences among RL modes reflect one realization of training.

In the PV-BES configuration, the same community adds 68.15 kWh of aggregate battery capacity and 32.07 kW of charge/discharge power across the eight prosumer households. Battery sizing and dispatch were determined upstream by a genetic algorithm minimizing annualized system cost, assuming roughly 91.4% round-trip efficiency, and the resulting state-of-charge trace is a fixed input that the settlement mechanism or RL policy cannot change.

Why This Matters

Impact on research. The paper supplies a clean counterexample to the assumption that learned pricing policies automatically beat simple rules. By holding the physical exchange, demand data, and PV portfolio constant, it isolates the pricing mechanism itself, and by including a fixed-parameter control within the same SDR-shaped family, it separates the value of the pricing rule's shape from the value of learning. The budget-balance argument — that single-price settlement keeps the whole grid price spread inside the community and therefore sets an upper bound on savings for any mechanism retaining an internal spread — is a structural insight that applies well beyond this case study.

Real-world applications

• Designing settlement rules for residential energy communities and cooperatives that want to share the value of local solar generation fairly among prosumers and consumers.
• Evaluating whether battery investment in a community is justified, and separating how much of the gain comes from the battery itself versus from the trading mechanism.
• Informing tariff and market design under net-billing schemes, where the export price can exceed the import price in some hours (384 of 8,784 intervals here).
• Choosing a pricing engine for a local energy market platform, where transparent rules may be preferable if they deliver equal or better outcomes than an opaque learned policy.

Industry relevance. Community energy platforms, aggregators, retail suppliers, and distribution system operators all need a defensible answer to how internal prices are set, since that choice drives participant payouts, the volume of energy cleared locally, and the resulting load on the distribution network. This paper's operational metrics — traded energy of 7,266 kWh and grid imports of 26,263 kWh for rule-based mechanisms versus 4,069 kWh and 29,459 kWh for RL-M in PV-only — translate directly into local congestion and self-sufficiency outcomes, and its household-level results show that an attractive community-wide number can hide a distribution that leaves one group with much less than the other.

Future Directions

• Test the rule-based benchmarks under storage directly. The rule-based mechanisms were evaluated only in the PV-only configuration, so the claim that they would retain their advantage under PV-BES rests on the budget-balance argument rather than on measured results. Re-running all six mechanisms with batteries would settle the question.

• Establish robustness across training runs. Results come from a single training run per mode and configuration with a fixed random seed, so the ranking of RL-M, RL-SDR-F, and RL-SDR-L reflects one realization. Multiple seeds and variance reporting would show whether the gaps — especially the €118 gap between RL-SDR-L and RL-SDR-F — are stable.

• Reconsider the price-quoting design that creates the internal spread. Since the spread is retained by the market operator and not redistributed, and since it accounts for the entire community-savings difference between RL-SDR-F and RL-SDR-L, a budget-balanced RL formulation with a single internal price is an obvious next design to test.

• Treat battery dispatch as a decision variable rather than a fixed input. The state-of-charge trace is exogenous and degradation-aware, so the storage contribution is bounded by the upstream sizing; coupling the pricing policy to battery operation could change both the savings and the household-level distribution.

• Address fairness explicitly. Community savings are the training reward, but the distribution across the 12 consumers and 8 prosumers varies widely between mechanisms, from near-even under SDR to prosumer medians above 100% under bill-sharing and MMR. Whether an allocation objective should enter the reward function remains open.

Target Audience

This paper is most useful to researchers working on local energy markets, P2P trading mechanisms, and RL applications in power systems; to energy economists and regulators assessing net-billing and community self-consumption schemes; and to practitioners designing or procuring pricing engines for community energy platforms. It also suits graduate students who want a worked example of benchmarking a learning-based control policy against transparent rule-based alternatives, including a non-learning control in the same policy family. Readers seeking detailed RL algorithm development, multi-seed statistical evaluation, or a re-evaluation of rule-based mechanisms under battery storage will not find those here.

Authors’ abstract

This paper compares rule-based and learning-based pricing mechanisms for peer-to-peer (P2P) electricity trading in residential photovoltaic communities. The rule-based benchmarks comprise bill-sharing as an ex post allocation mechanism, the mid-market rate, and supply-demand-ratio pricing. The reinforcement-learning (RL) formulation is implemented through a Deep Q-Network and evaluated under multiplier-based and learnable SDR-shaped pricing, with a fixed-parameter SDR variant as a non-learning control. Performance is assessed through community savings together with complementary financial and operational indicators. In the base PV-only configuration, the rule-based benchmarks outperform the best RL policy. With battery energy storage, evaluated for the RL policies only, community savings under the best RL policy increase from EUR 734.23 to EUR 978.52. Across the learning-based modes and in both configurations, SDR-shaped pricing outperforms the multiplier-based parameterization considered. The results indicate that rule-based pricing remains highly competitive wherever the two families are compared directly, and that storage substantially improves the learning-based outcomes under this accounting, while the distribution of benefits remains heterogeneous across households.