A Standardized Framework for Machine Learning in Power System Protection

AI summary

Overview

• Research area: Machine learning applied to electrical power system protection, with a focus on how such studies are evaluated, reported, and audited rather than on proposing a new model.
• Technical level: Intermediate. The paper combines protection-engineering concepts (relays, fault classification and localization, observability, decision windows, IEC standards) with machine-learning evaluation practice (splits, leakage, diagnostics, robustness), and is written to bridge the two.
• One-sentence scope: The paper proposes a seven-dimension standardized evaluation framework for machine-learning-based power system protection and instantiates it in a bounded case study on the public PROTECT-90 electromagnetic-transient benchmark for fault classification and fault localization.

Authors: Julian Oelhaf, Georg Kordowich, Paula Andrea Pérez-Toro, Christian Bergler, Johann Jäger, Andreas Maier, Siming Bayer (Friedrich-Alexander-Universität Erlangen-Nürnberg and Ostbayerische Technische Hochschule Amberg-Weiden). Published 2026-08-20, arXiv:2608.20181v1 [cs.LG].

What This Paper Is About

Machine-learning studies of power system protection increasingly report near-perfect scores, but the meaning of those scores depends heavily on the evaluation setting: the protection task, physical scope, available measurements, timing, target definition, preprocessing, and validation protocol often vary together and are only partly specified. The paper argues that when these choices are left implicit, results from different studies cannot be meaningfully compared, and a 99% score from one setup may correspond to a much simpler problem than a 90% score in another. Its goal is to turn evaluation design itself into an explicit, standardized part of the scientific contribution, and to demonstrate what such a framework reveals through a reproducible case study.

Key Contributions

1. A standardized, protection-specific evaluation framework. The framework structures a study into seven required dimensions: protection objective; physical system scope; observability and measurements; timing and decision windows; targets and valid samples; training and validation protocol; and evaluation outputs and diagnostics. It defines what must be specified, reported, and examined for a machine-learning protection result to be reproducible, comparable, and open to independent audit.
2. An interpretation logic treating performance as conditional evidence. Reported performance is tied to task formulation, physical scope, observability, timing, target construction, and validation design, so that apparent model capability is separated from differences in the underlying inference problem and aggregate scores are not read independently of the assumptions behind them.
3. A bounded, reproducible case study instantiation. The framework is demonstrated on the public PROTECT-90 electromagnetic-transient benchmark for fault classification and fault localization, under shared sensing, timing, sample-validity, and validation assumptions, spanning conventional protection, classical machine learning, task-specific deep learning, and a pre-trained time-series foundation model.
4. A standardized reporting package (defined as minimum required information per framework step), analogous in spirit to datasheets and model cards but documenting the evaluation setting of a protection study rather than a dataset or model; the framework is presented as a first methodological step toward certification-oriented assessment, not a formal certification procedure.

Main Findings

• Near-perfect classification under a fully specified setting. Under centralized sensing, simulation-metadata-aligned 20 ms windows, and episode-grouped validation on 9022 simulated episodes from a 90 kV double-line topology, a multi-layer perceptron (MLP) achieved a five-fold mean macro-averaged F1 score of 0.991 ± 0.001 for classification.
• Localization error is an order of magnitude less trivial. The same MLP reached a localization mean absolute error of 10.20 ± 0.25% of line length. Both standard deviations describe variation across the episode-grouped folds.
• A task-dependent performance asymmetry. Extending the decision horizon from 20 ms to 50 ms preserved the asymmetry between the two tasks, i.e., the relationship between classification and localization behavior was not an artifact of the shorter window.
• Observability affects the tasks differently. Reduced observability approximately doubled the MLP localization error but had little effect on classification.
• A conventional locator beat the learning locators under richer clean information. A synchronized two-ended conventional locator outperformed the learning locators under its richer clean information set.
• Clean accuracy did not predict robustness. Measurement degradation showed that clean predictive performance did not determine robustness.
• Existing reporting practice is structurally incomplete. In the authors' prior scoping review of 119 studies, only 16.0% used real-world data, 82.5% provided no data access, only 1.7% publicly released code or models, and even basic metadata such as sampling frequency was omitted in 52.1% of studies.
• An integration gap, not an absence of guidance. A structured comparison of representative guidance families (dataset documentation, ML reporting and validation guidance, protection reviews and scoping studies, protection-testing standards, and general AI risk and assurance guidance) shows that each addresses a complementary subset of the seven dimensions, whereas the proposed framework operationalizes all seven jointly. The comparison is explicitly described as an illustrative mapping, not a quantitative score.
• Onset conditioning is reclassified. The framework treats sample-validity rules, including onset conditioning, as part of the evaluated protection problem rather than an implicit preprocessing choice; prior guidance such as the IEC 60255-121 fault-inception-angle definition treats onset as a controlled test parameter, not a sample-validity rule.

Methodology in Plain English

The authors begin from a diagnosis rather than a model. They review how machine-learning protection studies are typically reported, note that task, scope, sensing, timing, targets, preprocessing, and validation tend to vary together, and conclude that this joint variation is what makes reported scores hard to interpret. From this they derive a seven-step framework that any study should specify before its numbers can be read: what protection function is being learned; what physical grid is represented; what measurements are available and how they are synchronized; what time budget and decision windows apply; how labels and regression targets are built and which samples count as valid; how data are split and how leakage is prevented; and what evidence—metrics, diagnostics, robustness analyses, runtime—must be reported.

They then show the framework in action rather than only describing it. The case study uses the public PROTECT-90 benchmark: 9022 simulated episodes generated in DIgSILENT PowerFactory from a 90 kV "Double Line" topology, with domain randomization over fault resistance, inception time, line characteristics, loading conditions, and external grid properties. The reference setting uses synchronized three-phase voltage and current measurements from 8 relays, ±80 ms around fault inception, 20 ms sliding windows with a fixed 5 ms stride, onset-conditioned 11-class fault classification (one non-onset class plus ten fault-type classes), and fault localization as a normalized fault position, with 5-fold grouped cross-validation by simulation episode.

Against that fixed setting, the authors run controlled analyses that vary one factor at a time: timing (extending the horizon to 50 ms), observability, measurement fidelity, fault-resistance distribution shift, stride, and hyperparameters. They compare conventional protection, classical machine learning, task-specific deep learning, and a pre-trained time-series foundation model under shared preprocessing and split design, so that differences can be attributed to model behavior rather than inconsistent evaluation conditions.

Why This Matters

Impact on research. The paper reframes evaluation design as a contribution in its own right. If adopted, it gives reviewers, editors, and authors a shared checklist that makes the effective inference problem visible before scores are compared, which is precisely what is needed to distinguish algorithmic progress from changes in task difficulty or information availability. It also connects protection-specific requirements (observability, synchronization, decision horizons, onset-conditioned sample validity) to established machine-learning validity requirements (leakage-aware splitting, preprocessing boundaries, metric and uncertainty reporting), an intersection that neither protection standards nor general AI-assurance guidance currently occupies on its own.

Real-world applications:

• Protection relay type-testing and certification. The framework supplies protection-specific evaluation dimensions that could feed a future certification-oriented assessment pathway for machine-learning protection functions, complementing practices such as IEC 60255.
• Utility procurement and vendor comparison. Utilities can require standardized reporting packages so that competing data-driven protection claims are stated under comparable task, sensing, timing, and validation conditions.
• Grids with inverter-based and distributed resources. As inverter-based generation, meshed and multi-terminal configurations, and hybrid AC–DC architectures expand the range of fault scenarios, deterministic threshold-based schemes are increasingly stressed; auditable evaluation of learned protection functions becomes more consequential.
• Independent audit and regulatory review. The reporting package documents data and code availability, configuration details, and restrictions affecting reproducibility, providing an inspectable artifact for third-party review.

Industry relevance. Protection is a safety-critical function where equipment is trusted only after standardized type-testing and third-party conformity assessment. Machine-learning approaches currently enter the field with no established basis for testing or auditing. The paper does not claim to offer formal certification or benchmark completeness, and it explicitly does not claim practical superiority over conventional protection—but it positions standardized evaluation as the procedural prerequisite for that trust to be established.

Future Directions

• Extending the framework beyond the demonstration setting. The case study covers fault classification and fault localization on one 90 kV double-line topology; fault detection, faulted-line identification, other voltage levels, HVDC and hybrid AC/DC systems, and wind-integrated configurations remain open.
• Moving from methodological step toward formal certification. The paper positions the framework as a first step toward certification-oriented assessment without constituting a formal certification procedure; how the seven dimensions would map onto concrete type-testing requirements and conformity assessment is unresolved.
• Closing the real-data gap. The case study is simulation-based, reflecting the scarcity of publicly accessible real-world protection datasets with sufficiently detailed waveform recordings and labels; field-recorded validation and documentation of data-generation fidelity are natural next steps.
• Turning robustness into a first-class result. Since clean predictive performance did not determine robustness under measurement degradation, and observability changes affected localization far more than classification, the authors leave open how robustness, degraded-sensing conditions, and end-to-end latency (as opposed to short decision windows alone) should be reported and compared systematically.

Target Audience

Protection engineers and relay specialists who want data-driven methods evaluated under operationally meaningful assumptions; machine-learning researchers entering the power systems domain who need to know which evaluation choices actually define the inference problem; reviewers, editors, and standards or certification bodies assessing reproducibility and auditability of machine-learning protection claims; and utility or vendor engineers who must compare competing protection approaches on a like-for-like basis. Readers looking for a new protection algorithm will not find one here—the paper's value is in the evaluation structure it imposes.

Authors’ abstract

Studies of machine-learning-based power-system protection increasingly report near-perfect scores, yet the meaning of those scores depends strongly on the evaluation setting. Protection task, physical scope, measurements, timing, targets, preprocessing, and validation often vary jointly and remain incompletely specified. This paper proposes a standardization-oriented framework that treats evaluation design as part of the scientific contribution. It defines seven required study dimensions: protection objective, physical scope, observability, timing and decision windows, targets and sample validity, validation protocol, and evaluation outputs. The framework is instantiated in a bounded case study on the public PROTECT-90 electromagnetic-transient benchmark, comprising 9022 simulated episodes from a 90 kV double-line topology, for onset-conditioned fault classification and localization. Under centralized sensing, simulation-metadata-aligned 20 ms windows, and episode-grouped validation, a multi-layer perceptron (MLP) achieved a five-fold mean macro-averaged F1 score of 0.991 +/- 0.001 for classification and a localization mean absolute error of 10.20 +/- 0.25% of line length (mean +/- std across episode-grouped folds). Extending the decision horizon to 50 ms preserved this task-dependent performance asymmetry, while reduced observability approximately doubled the MLP localization error but had little effect on classification. A synchronized two-ended conventional locator outperformed the learning locators under its richer clean information set, and measurement degradation showed that clean predictive performance did not determine robustness. The framework turns evaluation assumptions into explicit, reproducible evidence and provides a basis for more comparable, auditable evaluation and future certification-oriented assessment of machine-learning protection functions.