PGLearn - An Open-Source Learning Toolkit for Optimal Power Flow

We thank the reviewer for their close review and insightful comments. Detailed answers to the reviewer's questions follow.

The input data is generated using a uniform sampling procedure, but in the future to extend the work for Unit Commitment or multi-period OPF, time series data should also be incorporated. For instance, the recent paper "PowerGraph: A Power Grid Benchmark Dataset for Graph Neural Networks" provides an OPF dataset that uses time series data to model input demand.

We thank the reviewer for pointing us to this recent paper. We agree that capturing temporal structure is an important avenue for generating more realistic datasets, and we intend to do this in future versions of PGLearn. Please refer to our general response for additional details.

Could you provide comments on the results of benchmarking the datasets using the provided metrics across different machine-learning methods? A leaderboard would be the easiest way to access and interpret these results.

We thank the reviewer for their suggestion to create a leaderboard. While we unfortunately do not have the resources (both time-wise and computing-wise) to re-implement and benchmark all existing ML methodologies for Optimal Power Flow problem, we agree that a public leaderboard would provide valuable information to the community. We hope to develop a community-driven leaderboard where researchers can submit their results to faciliate the integration of multiple methodologies, similar to, e.g., a Kaggle competition.

We thank the reviewer for their close review and valuable feedback. Detailed answers to the reviewer's questions follow.

Algorithm 1, the notation of what you return could be misleading as you used both for referenced demand and sampled demand. It looks as if you are returning the same reference value at the end of the algorithm.

We thank the reviewer for their careful review of the notation used in the paper. We have updated the notation in Algorithm 1 to avoid confusion.

The supplementary material is not self-explanatory for someone outside of the OPF domain; I can understand each file from what I read in your paper's appendix, but you could have added basic instructions. Without instructions, it is just a folder with h5 files.

For a work like yours, you should've added an anonymized version of your code to assess how you implemented what you described only with words in the main paper

We thank the reviewer for raising this point, and have included anynomized code repositories to the supplemental material. Please review the README files at the root and within each repository for basic usage instructions, and note that the source code is documented via docstrings. The official repositories (which cannot be shared under ICLR anonymity rules) also provide live documentation.

We thank the reviewer for their careful reading of our paper, and insightful feedback. Answers to the reviewer's specific questions follow. The response is split into two comments due to the character limit.

While the datasets aim to capture real-world conditions, unforeseen variability in actual grids may still limit the applicability of the models trained on these datasets.

We thank the reviewer for raising this point, as we recognize the importance of capturing real-world conditions as closely as possible. As discussed in the paper, the vast majority of existing works consider only uncorrelated load-level noise, which yields a very narrow range of total demand across samples (as shown by Figure 2). In contrast, our approach explicitly captures total demand fluctuations by introducing a global scaling factor (denoted by $b$ in Algorithm 1). Figure 2 demonstrates that this strategy not only yields a broader range of total demand, but also results in datasets with more complex dynamics. We believe this is a meaningful step towards more realistic datasets, as this mechanism better captures diurnal / seasonal variations in total demand. Nonetheless, we agree that more sophisticated correlation structures derived from real-world data are valuable for future research, and intend to include this aspect in future releases of PGLearn.

It only focus on OPF problems, which may not be fit for the general audience of the ML community

We thank the reviewer for this feedback regarding the breadth of PGLearn. We would like to point out that there has been significant interest from the ML community in developing new methodologies for optimization problems that arise in power systems, as demonstrated by the vast body of existing work (see survey papers we cite and the references therein). Furthermore, OPF has proved to be a popular testbed to develop general-purpose ML methodologies for optimization and constrained learning in general, including: DC3, Primal-Dual Learning, Gauge Mapping, Homeomorphic projections, Dual optimization Learning. In that context, PGLearn provides large-scale datasets that comprise multiple OPF formulations spanning three classes of optimization problems: ACOPF (nonlinear, non-convex), SOC-OPF (nonlinear, convex) and DCOPF (linear, convex). The collection also provides user-friendly, vectorized data structures that are readily usable by ML researchers, without needing a complete understanding of power systems dynamics. Therefore, we believe that PGLearn has the potential to support not only research in ML for OPF problems, but broader topics such as ML and optimization and constrained learning as well.

For my best understanding, the paper mainly focus on the dataset introduction, it does not deliver the technical advances in the search field

The main contributions of this paper are 1) the PGLearn dataset, 2) the standardized evaluation methodology, and 3) the accompanying open source data-generation and evaluation code. We believe these contributions align with the scope and objective of the "datasets and benchmarks" track which this paper has been submitted to.

It is also useful to add a set of adversial samples in the dataset for robustness testing under the worst case scenario

We thank the reviewer for raising this point regarding adversarial samples and robustness. To the authors knowledge, adversarial samples are typically considered "adversarial" against a particular model. We are curious what exactly the reviewer has in mind regarding adversarial OPF samples on the dataset level.

The authors are suggested to include a general review on the ML for OPF papers, e.g., how ML are generally applied to OPF problems. There have been several literature review papers on it.

We thank the reviewer for their suggestion. The paper's introduction and related work sections (Section 1 and Section 1.2) provide an overview of existing methods and refer the reader to the relevant literature review papers. We have added a citation to an earlier survey paper on machine learning and optimal power flow: Hasan, F., Kargarian, A., & Mohammadi, A. (2020). A survey on applications of machine learning for optimal power flow. 2020 IEEE Texas Power and Energy Conference (TPEC). Space limitations prevent us from providing a self-contained literature review in the paper, which is why we elected to cite existing surveys on the topic.

We thank the reader for their close review of our paper; it has helped us further improve PGLearn. Detailed answers to the reviewer's questions follow. The response is split into two comments due to the character limit.

Could the authors point the link to the anonymousrepo? And in these two repos, are the proposed four accuracy metrics and computation metrics realized?

We replaced the names of our official repositories with "AnonymousRepo1" and "AnonymousRepo2" to comply with anonymity rules of ICLR, since that information would have enabled to de-anonymize our submission. Please find anonymized source code in the updated supplemental materials. This code indeed contains the implementation of the various performance metrics reported in our paper. Naturally, the final version of the paper will contain the name and full URLs of code repositories and datasets.

There is no inclusion nor analysis of contingencies in OPF problems. Essentially the introduction mentions renewable uncertainties multiple times, while the proposed method does not address such uncertainty issues.

We thank the reviewer for raising this point. We would like to point out that PGLearn includes instances with so-called N-1 cases, wherein an individual generator or line is unavailable; this is similar to OPFData. Furthermore, uncertainty in renewable production is typically captured via the so-called "net load," i.e., demand minus renewable generation.

The PGLearn module is limited to implementations of few methods by similar group of authors. A more thorough literature review and inclusion of methods are recommended: Zhang, Ling, Yize Chen, and Baosen Zhang. "A convex neural network solver for DCOPF with generalization guarantees." IEEE Transactions on Control of Network Systems 9, no. 2 (2021): 719-730. Zhou, Min, Minghua Chen, and Steven H. Low. "DeepOPF-FT: One deep neural network for multiple AC-OPF problems with flexible topology." IEEE Transactions on Power Systems 38, no. 1 (2022): 964-967. Owerko, Damian, Fernando Gama, and Alejandro Ribeiro. "Unsupervised optimal power flow using graph neural networks." In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6885-6889. IEEE, 2024.

We thank the reviewer for their valuable suggestions. The goal of our work is to provide standardized datasets and performance metrics, together with a unified interface that enables researchers to evaluate their method on PGLearn. An exhaustive re-implementation and benchmark of all existing approaches in the literature would require far more (time and computing) resources than we have available.

In Introduction, "previously intractable applications such as real-time risk analysis", yet in this work, there is no discussion on how ML can help with OPF risk.

The mention of real-time risk analysis in this sentence refers to the work of Chen, W., Tanneau, M., & Van Hentenryck, P. (2024). Real-time risk analysis with optimization proxies. Electric Power Systems Research, 235, 110822. This work has shown that large-scale Monte-Carlo simulations involving the resolution of multiple OPF problems become computationally tractable by leveraging ML-based proxy models. Naturally, the accuracy of these simulations directly depends on the accuracy of the underlying ML models.

The PGLib-OPF seems to be an important reference. Could the author explain more on "PGLib-OPF only provides a single snapshot per grid"? Does that mean only one data sample is provided for each power grid model?

We thank the reviewer for raising this point. Indeed, PGLib provides only one problem instance for each power grid. The PGLib collection was released in 2019 to "benchmark AC optimal power flow algorithms". It now comprises 66 instances of optimal power flow problems, each corresponding to a different power grid. Most cases are artificial power grids (e.g. IEEE standard test cases), some are snapshots of real transmission systems, e.g., the RTE and Pegase cases. To support the ML community in working on OPF problems, our work thus provides open-source data-augmentation code, standardized datasets generated using this code, and open-source code to evaluate the performance of ML models.

I appreciate the authors' further clarifications. But the real computation burden for researchers is not necessarily coming from solving optimization for large grid OPF offline. Rather it is coming from the larger machine learning model training, which this work does not shed any light on. In addition, the claim “Our work is the first to provide, together with open datasets containing large-scale OPF instances, data-generation and model evaluation code” is also not exact, as works such as OPF-Learn [1] already did that, which limited this work's novelty.

And considering the lack of realistic data rather than just sampling based on statistical distribution, the existence of standard grid testing library, and the absence of enough benchmarked ML for OPF algorithms to compare performance, I would still doubt this work's novelty and contributions. And just like the authors suggested, it would be meaningful to evaluate current algorithms' performance under standardized datasets and metrics, while this work falls short of that.

Reference [1] Joswig-Jones, Trager, Kyri Baker, and Ahmed S. Zamzam. "Opf-learn: An open-source framework for creating representative ac optimal power flow datasets." In 2022 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), pp. 1-5. IEEE, 2022.

We thank the reviewer for their close reading of our paper and for their insightful comments. The response addresses the raised points individually:

There are not many systems implemented in the library and most of them are large systems. Researchers may want to work on smaller systems for proof of concept or visualizing the model behaviour. It would be nice to add them.

We agree with the reviewer that including smaller cases can be valuable for preliminary studies. We have amended the list of benchmarks to include additional small cases 14_ieee, 30_ieee, 118_ieee, and 300_ieee, as well as large cases 13659_pegase and midwest24k (as suggested by Reviewer EiRM). The full case statistics and references are included in the general response.

While the proposed data augmentation method generates wider range of demand dynamics than sampling random factor and multiplying the base load, I am not sure this approach captures the global dynamics. The demand characteristics can differ widely within the hours of a day or among different seasons, which can result in different constraints to bind at the solution. This is mentioned as a limitation, but I think more detailed demand scenarios should be considered to train more reliable NN models.

We thank the reviewer for raising this important point, as we agree with the importance of capturing global dynamics. As noted in our paper, the vast majority of existing works consider uncorrelated, bus-level perturbations, which yield a very narrow range of total demand (as illustrated by Figure 2 in our paper). In contrast, our approach explicitly captures variations in total demand by introducing a global scaling factor (denoted by $b$ in Algorithm 1). Figure 2 demonstrates that this strategy not only yields a broader range of total demand, but also results in datasets with more complex dynamics. We believe this is a meaningful step towards more realistic datasets, as this mechanism better captures diurnal / seasonal variations in total demand. We nonetheless agree that more granular correlation structures are valuable for future research, and intend to include this aspect in future releases of PGLearn.

Thanks to the authors for their response and adding smaller test systems to the library.

About capturing the global dynamics, it is true that majority of existing studies consider bus level perturbation. For some cases, this approach leads to models that perform well on test set but still cannot generalize outside of the training distribution.

I agree that the proposed approach is a meaningful step towards the realistic datasets and it is a valuable contribution, but the wording "global" in the abstract ( "PGLearn implements realistic data generation procedures that capture both global and local variability") may not be accurate in this context. I have doubts that if researchers train their models using the proposed library, the resulting model can predict with the same performance in case of dramatic changes in load characteristics, such as pandemics or natural disasters. However, the rest of the appearances of the word (e.g. "global scaling", "global range") are more meaningful within the context.

Overall, this does not impact my decision on the value of the study.

We thank the reviewer for their time reviewing our paper; the feedback has helped us further improve the paper. The response will address the raised points individually:

The instances on up to 9241 buses (the Pegase project instance) are smaller than those of the Deepmind OPFData (Lovett et al., 2024) with instances on up to 13,659 buses.

We thank the reviewer for their suggestion to include more large-scale cases in PGLearn. We have amended the list of benchmarks to include additional large cases 13659_pegase and midwest24k, as well as small cases 14_ieee, 30_ieee, 118_ieee, and 300_ieee (as suggested by Reviewer M2Kx). The updated version of the benchmarks table including the full case statistics can be found in the general response. Finally, we would like to point out that, compared to DeepMind's OPFData, we provide 1) both primal and dual solutions, 2) solutions for multiple OPF formulations, 3) a more realistic data generation procedure that covers a broader range of total demand, and 4) open-source data generation and model evaluation code.

While considering correlated demand perturbations may improve the realism somewhat, the authors should ideally consider real-world time-series capturing diurnal variations, or similar. Likewise, they could consider time-varying production limits, which are readily available to all market participants in Europe via the ENTSO-E Transparency Platform (https://transparency.entsoe.eu/).

We thank the reviewer for this insightful feedback, and we agree that better capturing temporal structure is important for further research. The inclusion of a global perturbation in our data-generation methodology allows to capture a broader range of total demand which, in turn, better captures diurnal variations. We believe that this approach yields a meaningful improvement over existing publicly-available datasets. We intend to capture temporal structure more explicitly in future versions of the PGLearn collection.

We also thank the reviewer for pointing out the ENTSO-E transparency platform. This platform releases time series information at the regional/zonal level, which must therefore be disaggregated to obtain bus-level information; how to accurately do this disaggregation is an important direction for future research. We would also like to point out that ENTSO-E does not release detailed grid topology information, e.g., line impedance, generator capacities, individual load profile, etc. which is typically protected.

The supplementary material contains no code, so the code could not checked for reproducibility or usefulness.

We thank the referee for pointing this out. We have included anonymized versions of the repositories in the updated supplemental materials. Please note that although our code will be available under an open-source license, we cannot directly share those official repositories as it would violate ICLR anonymity rules.

Dear EirM, as you are one of the reviewers who expressed Confidence level 5, I can assume you are a subject expert, and I am sure you can take some time to provide more detailed feedback to the authors; I am sure they would appreciate that as their main objective is working on convincing you that their work is worth the acceptance.

I can notice from your review that although your confidence is the best, you didn't spend much time writing this review and didn't follow the guidelines of the sections (you addressed weaknesses in the summary space), preventing the authors from understanding the details that could make you reconsider your rating, which is the whole idea of the peer review process.

My contribution to the discussion here is as follows:

• I believe that the contribution is not centered around the size of the instances, so I don't consider it a strong factor to compare against works like Google DeepMind OPFData.
• I agree that real-life perturbations would be ideal for the authors. Still, the contribution from the authors lies in considering the correlation in the sampling strategy, which is an advance and still a contribution. It is, in a way, a modeling effort that, although simplistic, makes the case more realistic. Reviewer 19cY made more detailed points on this decision from the authors; you might want to look.
• I agree with the authors that including real-time data from the ENTSO-E Transparency Platform could be a different problem and have other limitations (like the access to topologies), so it might not be a strong point the authors could address.
• I agree with the lack of code on the supplementary material; you will notice that they added an update.

Thanks in advance for your attention!