PowerPM: Foundation Model for Power Systems

Thank you for your constructive and detailed comments. Responses to specific comments are listed below.

1. W1&Q7: Compare their framework with some state-of-art techniques

Thank you for your advice. Since ETS is essentially a time series, we choose the time series modeling model as the baseline, and all of its models are tested in the field of power systems in the original article.

Our baseline includes not only the SOTA model of supervised learning, but also the SOTA model of self-supervised learning. With the application of large language models in time series, we also select three LLM enhanced Time series models of SOTA. See section "Experiment 3.1" and Appendix 4.1 for a description of baselines.

Therefore, our experiment included three categories of SOTA models, and conducted sufficiently complete experiments, as shown in Table 1.

2. W2&Q1: Discussion on model interpretability

Thank you for your insightful comment on the interpretability of the PowerPM model. We acknowledge the importance of model interpretability and its impact on user trust and model adoption. Your feedback is valuable in guiding us to enhance the discussion on this aspect.

In terms of model architecture, PowerPM consists of a temporal encoder and a hierarchical encoder, whose design goals are clear and public. Time encoders utilize Transformer encoders to capture temporal dependencies in time series and learn to predict future time points through mask ETS modeling strategies in the self-supervised learning framework.

Hierarchical encoders use graph neural network technology, such as R-GCN, to simulate complex relationships between different hierarchy. This representation of hierarchical correlations is essential for understanding power system dynamics from both macro and micro perspectives.

Our self-supervised learning framework further enhances the model's understanding of temporal dependence and differences across ETS windows. This not only improves the performance of the model, but also gives us a way to explain how the model learns these dependencies.

3. W3&Q2: Explanation on Hyperparameter Selection

Thank you for pointing out our problem. Due to the large number of parameters and huge pre-training data, and the long pretrain time of the model, considering that the existing foundation model work has not carried out the super-parameter search, for the sake of efficiency, the super-parameter in the pre-training stage is set by default with reference to the existing work.

3. W4&Q4: Scalability and Deployment Considerations

Thank you for raising these doubts. We have successfully deployed a lightweight version of PowerPM on the national grid for load forecasting.

The main challenge encountered during the deployment process is that the inference resources of the National Grid are limited, resulting in low efficiency of the model operation and can not get real-time results. The solution is as follows:

During the deployment phase, we trained models with different parameters for different tasks, including 120M and 250M, to achieve a balance of efficiency and accuracy. It is deployed on the grid's six NVIDIA 3090 for inference. For functions that are called more frequently, such as user load forecasting, we take into account efficiency factors and use a 120M model for inference. For functions that are called less frequently, such as area load forecasting, a 250M model can be used for inference, thus guaranteeing accuracy.

At the same time, this scalability is very free, and the architecture of the model can be modified according to the number of parameters you want to adjust.

4. W5&Q5&L2: Ethics statement and social impact

The data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. The data has been effectively licensed by State Grid Corporation of China in Zhejiang province, all user-related information is encrypted, and all downstream tasks are only used for State Grid supply planning and internal analysis. The social impact is shown as Appendix G.

5. Q3: Discussion on Generalization to Unseen Domains

Thank you for asking this question. Since PowerPM is a foundation model proposed for the power system, it cannot be generalized to data sets in the unknown field used in the experiment, but it can be generalized to any data set in the power system, which has been verified in the paper. We conducted training on private data sets and generalized it to four public datasets. The result is shown in Table 2.

If the data set has a hierarchical relationship, the graph structure can be constructed. If there is no hierarchy (that is, the graph structure is an isolated graph), PowerPM can model normally without any modification. Therefore, PowerPM has a strong ability to adapt to new data distributions and scenarios in power system.

6. Q6: Future Directions and Extensions

Thank you for your interest in the future direction of our work. The future development of PowerPM is divided into the following two stages:

$\bullet$ Stage 1: Deploy as many downstream tasks as possible to the National Grid for landing. This will make grid data analysis more intelligent and automated, and assist grid workers in decision-making, reducing costs and increasing efficiency.
$\bullet$ Stage 2: Currently, we only consider the macro-level interaction modeling. In the future, we will consider more factors, such as power grid topology, Kirchhoff's law and other constraints. To enhance the modeling process of ETS data to achieve a more general representation. This enhances downstream task performance and provides a more unique perspective for modeling ETS.

Above is the future development direction of our work, thank you for your attention and support to our work.

If you have any other concerns, please let us know what we can do to address any concerns you may have. Thanks for your thoughtful feedback again!

Dear Reviewer,

Please reply to the rebuttal.

AC.

Thank you for your constructive and detailed comments. We apologize for the imprecise claims in certain parts of our manuscript. We will reconsider the claims to be more precise. We hope that the responses below could address your specific comments.

1. W1: Clarify where the performance of the model comes from and the technical contribution of the model

Thank you for your valuable comments. Please refer to Global Response for answers to these questions.

2. W2&L1: Only looks into a very general forecasting problem and disscuss generalization

Most of methods forecast on a single or small number of instances and have poor generalization. Due to the large difference in the electricity consumption of different users, the effect decreases significantly when it transfers to other regions. Secondly, most studies do not consider the correlation between hierarchies in ETS modeling. Our experimental results show that PowerPM achieves SOTA on the four public datasets, demonstrating strong generalization of the forecasting task and modeling hierarchical relationships can improve the effect of forecasting tasks.

At the same time, our model is not limited to forecasting tasks, but can also perform tasks such as missing value imputation, electricity theft detection and so on. Through large-scale pre-training, PowerPM can successfully generalize to 44 power system tasks, and has a strong generalization ability, which is fundamentally different from the current end-to-end developed models for many power system tasks

3. W3: The grid stability notion is misleading

As you said, stability notion is with respect to either frequency or voltage, and transient states of the system. And these tasks are also summarized around these aspects:

$\bullet$ Electricity imputation:The missing load value will lead to misleading scheduling decisions, which will lead to the instability of the system voltage and destroy the grid stability[R1].
$\bullet$ Clock anomaly detection:Clock anomalies will lead to clock frequency instability, resulting in synchronization errors, affecting the coordinated operation of power grid equipment, which will lead to instantaneous state out of control, affecting the grid stability[R2].
$\bullet$ Electricity theft detection: The behavior of stealing electricity will lead to the voltage change of local power grid, affecting the normal consumption of other users, which will also affect the grid stability[R3].

Specifically, these three tasks are all about the frequency or voltage and the itransient states of the system, so we summarize them as grid stability.

4. W4: Disscuss on computational costs

Thank you for your valuable comments. Please refer to Global Response for answers to these questions.

5. Q1: The effects of contrastive learning

Contrastive learning is used to capture the differences between different ETS windows. The results of Ablation experiments can be referred to in section 3.4 "Ablation Study", which shows that contrastive learning can effectively improve the performance of downstream tasks related to classification and anomaly detection. the related literature points that contrastive learning has assumed the downstream applications to be classifications[R4].

6. Q2: Elaborate the settings of the freeze version

We are sorry that we don't provide clear guidance in the manuscript so that you missed this part. The details of frozen version of PowerPM can be find in Appendix D.1, "Partial Fine-tuning" section.

In the P-FT (Partial Fine-tuning) setup, for different tasks, we also introduce different head $H$ on the top of pre-trained encoder $f(.)$. For forecasting and imputation tasks, we use a prediction $H_l$ head to map prediction points or reconstruction points from $\mathbf{z}_i$. And for anomaly detection and classfication tasks, a classifier $H_c$ on top of the pre-trained encoder $f(.)$.
During the whole finetune process, we keep the parameters of $f(.)$ fixed. Only the head is fine-tuned in this setup.

7. Q3: Explain the major information conveyed in this Figure 4

As you can see from the figure, the yellow bar chart indicates the electricity loss from actual orderly reductions in electricity use, whereas the blue bar chart represents the electricity loss from scheduled orderly reductions in electricity use. As observed from the figure, the scheduled orderly reductions predicted by PowerPM approach the energy loss resulting from actual reductions, which is sufficient to provide guidance for the planning of power systems, demonstrating the effectiveness of PowerPM. A detailed description can be found in Appendix B.

If you still have any other new concerns, we would be eager to know what we can do to address any questions or concerns you may have. Thanks for your thoughtful feedback again!

Reference:

[R1] Wang M C, Tsai C F, Lin W C. Towards missing electric power data imputation for energy management systems[J]. Expert Systems with Applications, 2021, 174: 114743.

[R2] Zhang H, Wang Q, Li Y, et al. Clock Anomaly Detection Method of Power Quality Monitoring Device Based on Voltage Sag[C]//2021 IEEE 2nd China International Youth Conference on Electrical Engineering (CIYCEE). IEEE, 2021: 1-6.

[R3] Depuru S S S R, Wang L, Devabhaktuni V. Electricity theft: Overview, issues, prevention and a smart meter based approach to control theft[J]. Energy policy, 2011, 39(2): 1007-1015.

[R4] Liu X, Zhang F, Hou Z, et al. Self-supervised learning: Generative or contrastive[J]. IEEE transactions on knowledge and data engineering, 2021, 35(1): 857-876.

Dear Reviewer,

Please reply to the rebuttal.

AC.

Thank you for your constructive and detailed comments. Responses to specific comments are listed below.

1. W1: About source of the model performance

Thank you for pointing out that our description may confuse the reader. We have made a detailed reply in Global Response , please check and wish it can answer your confusion.

2. Q1: About computational resources used

We are sorry that we don't provide clear guidance in the manuscript so that you missed this part. The computational costs and training process can be found in Appendix D.1. All the experiments are repeated five times, conducted on a Linux system with 2 CPUs (AMD EPYC 9654 96-Core Processor) and 8 GPUs (NVIDIA Tesla A800 80G) for about 8 days.

3. Q2: Codes related to the proposed pre-training method

Sorry we didn't have a clear description of the submitted code, so that you missed this pre-training method.

The complete code has been placed in the supplementary material and has been uploaded at the time of the first submission of the manuscript. The file structure is as follows: (Hidden files have been ignored)

.
├── configs
├── environment.yaml
├── logs
├── Makefile
├── pyproject.toml
├── result
├── scripts
│   └── pretrain
│   └── run.sh
├── src
│   ├── models
│   │   ├── powergpt_module_pretrain.py
│   │   └── PowerGPT.py
│   └── powergpt_pretrain.py
└── tests

The entrance is in the process of the pretraining is ./ scripts/pretrain/run.sh, the script will execute python ./src/powergpt_pretrain. py. Meanwhile, the ./src/models/powergpt_module_pretrain.py file contains the pre-training process, where loss_cl and loss_mse are the loss functions corresponding to the two pretraining tasks. And ./src/models/PowerGPT.py is the code file of our model backbone, which contains all the implementation code for the model and the detailed procedure for the pre-training tasks. That's all the relevant code and files for our pre-training.

4. Q3: Our planned steps for the dataset and pre-trained model release

We apologize for the temporary unavailability of the dataset, because it involves highly sensitive user electricity usage data and personal privacy concerns. But we will release our pre-trained model.

In the future, in order to make this dataset a scientific research tool and better serve the research community, we plan to promote the release of our dataset in the following three stages:

$\bullet$ Stage1: We plan to publicly release the pre-trained models with different scale in 250M、128M、64M and 35M, shortly after our work is accepted. And we also release the four public datasets. This will enable other researchers to not only replicate our model’s experimental results, but also utilize these models for other research of interest.

$\bullet$ Stage2: We will actively communicate with the State Grid Corporation of China in Zhejiang province and aim to release the raw data of a portion of instances by the end of the year, to support further research efforts. The same as the work in our manuscript, these releases will be also conducted in compliance with ethical review requirements.

$\bullet$ Stage3: In the future, we will explore the possibility of releasing the full dataset following approval of the relevant ethical review, to allow researchers to use the large-scale dataset for more research.

5. L1: Ethics statement

The data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. The data has been effectively licensed by State Grid Corporation of China in Zhejiang province, all user-related information is encrypted, and all downstream tasks are only used for State Grid supply planning and internal analysis.

If you still have any other new concerns, we would be eager to know what we can do to address any questions or concerns you may have. Thanks for your thoughtful feedback again!

Dear Reviewer,

Please reply to the rebuttal.

AC.

We thank the reviewer for all the insightful comments. We apologize for the imprecise claims in certain parts of our manuscript and misunderstandings caused by our writing. Responses to specific comments are listed below.

1. W1: Several places are missing space between words

Thank you for pointing out our writing problems. We are sorry that we have caused you discomfort in reading due to our negligence. We've gone over The spelling issues and missing space, The updates are as follows:

$\bullet$ in line 77 of the manuscript, "Model (PowerPM)," will be changed into "Model (PowerPM) ,"
$\bullet$ in line 77 of the manuscript, ”about 250Mparameters“ will be changed into ”about 250M parameters“.
$\bullet$ in line 763 of the manuscript, "pre-train on Load data." will be changed into " pre-train on load data."
$\bullet$ in the caption of figure 1, "(d) Various tasks in power systems" will be changed into "(d) Various tasks in power systems."

2. W2: Compare their framework with some state-of-art techniques

Thank you for your advice. Since ETS is essentially a time series, we choose the time series modeling model as the baseline, and all of its models are tested in the field of power systems in the original article.

Our baseline includes not only the SOTA model of supervised learning, but also the SOTA model of self-supervised learning. With the application of large language models in time series, we also select three LLM enhanced Time series models of SOTA. See section "Experiment 3.1" and Appendix 4.1 for a description of baselines.

Therefore, our experiment included three categories of SOTA models, and conducted sufficiently complete experiments, as shown in Table 1.

3. Q1: Whether considered the power systems constraints

Thank you for your suggestion, we are sorry that we did not take into account the power systems constraints, such as Kirchhoff's circuit laws, etc.

Because all the data we currently obtain is provided by State Grid Corporation of China in Zhejiang province, it is the electricity consumption data after desensitization, which only contains the instance data recorded in the terminal meter and sensor. Its minimum granularity only sinks to the user unit and does not include complex circuit topological relationships within or between users. Therefore, we do not consider the constraints of the power network topology here, and only model ETS according to the physical level constraints in the real world, which can achieve quite superior results.

But it's worth noting, your suggestion is very promising and meaningful, because considering more constraints can better model the power system. In the future, we will consider including power network structures, such asKirchhoff's circuit laws and other factors to model ETS. Please keep paying attention to our work and thank you again for your suggestions.

4. Q2: Whether considered of including weather time series data

Yes, as you said, we have considered weather time series data. Please refer to lines 135-142 in the "Method" section for the detailed description and figure 2(b) for the legend.

We crawled the local weather and temperature on the public website, called as exogenous variables, and assigned the learnable parameters according to its different values and mapped to the embedding table. Each ETS windows has a corresponding sequence of exogenous variables. When the temporal encoder models ETS, the corresponding exogenous variables will look up the corresponding representation in the embedding table index, and finally add and fuse to obtain the ETS representation enhanced by external variables.

Thank you again for your thorough consideration. The ablation experiment results are shown in Table 7 in the Appendix. After adding the weather and other exogenous variables, the performance of the model will indeed be improved to a certain extent, especially the Solar generation forecasting which is greatly affected by environmental factors.