Expand and Compress: Exploring Tuning Principles for Continual Spatio-Temporal Graph Forecasting

---

Q3: Long-term effectiveness: Will similar trends be observed in non-small-sample scenarios? Is training separately for each period likely to achieve better performance?

• We apologize for any unnecessary misunderstandings. This issue likely arises from our attempt to be fair and compress enough comparative information in Table 1. Below, we provide a comparison between the Retrain-ST method (which trains separate models for each period) and our method on the PEMS-Stream dataset, comparing performance over avg. 12-step predictions across multiple periods (averaged over five random runs):

• For large datasets, performance varies across periods mainly due to noise and difficulty levels in the current period’s dataset. As for retraining separately, it actually doesn’t perform well, as explained in the previous response. Also, retraining typically requires starting from scratch. By inheriting previous weights, we can significantly accelerate the optimization process, as shown in the table below, where we compare the average training time per period between our method and retraining each period separately (averaged over five random runs on the PEMS-Stream dataset):

---

We greatly appreciate your valuable feedback, and we will incorporate these detailed discussions into the final revision of the paper. We hope the above answers help address your concerns. If possible, we kindly request you to reconsider raising the score. If you have any further suggestions, we would be more than happy to discuss them and make necessary improvements to the paper.

Best regards,

All authors

Dear Reviewer YV6D,

We sincerely thank you for taking the time and effort to provide valuable feedback on our paper. We apologize for any misunderstandings caused. We have carefully considered each of your points and have addressed them one by one.

---

Q1: The design of the prompt pool is not clearly explained, and there is a lack of clarity on how the system handles the integration of new sensors in a dynamic environment.

• We are sorry to hear that. A gentle reminder: you may have missed the pseudocode and detailed description of our algorithmic workflow in Appendix B.
• Furthermore, the way the system handles sensors in dynamic environments is precisely the role of the prompt parameters added to our prompt pool. As explained in Section 4.1, through detailed empirical observations and theoretical analysis, we derive principle 1, which shows that the prompt pool can continuously adapt to the heterogeneity of spatio-temporal data generated by new sensors in dynamic environments.
• We sincerely hope these clarifications help clear up any misunderstandings.

---

Q2: The evaluation lacks comparison with models trained separately for each period. It is necessary to establish performance upper bounds by comparing with the scenario where models are trained separately for each period.

• We apologize for the confusion. A gentle reminder: you may have missed our description of the Retrain-ST method, which is precisely designed for the scenario where separate models are trained for each period. Regarding why its performance is not satisfactory, we have provided a detailed analysis in Section 5.2 of the paper. However, we will summarize and address it here for your quick understanding:

  • The Retrain-ST method tends to show unsatisfactory results mainly because it relies solely on limited data to train period-specific models, without effectively utilizing the historical information from the pre-trained model. In continuous spatio-temporal graph scenarios, underlying spatio-temporal dependencies are shared, and more historical training data typically helps the model achieve better performance.

• If your concern is related to the stacking of all historical data to train the model, we would also like to clarify that while stacking all historical data may sound reasonable at first, it is actually impractical and represents a misunderstanding of continuous spatio-temporal graph learning. As pointed out in the last sentence of the introduction: “Due to computational and storage costs, it is often impractical to store all data and retrain the entire STGNN model from scratch for each time period." Therefore, the motivation behind all continuous spatio-temporal graph modeling methods is as follows:

  • (Training and Storage Costs): Storing all historical data and retraining is associated with unacceptable training and storage costs. Training costs are easy to understand, but storage costs are significant because the model is usually only a fraction of the size of the data (e.g., in the PEMS-Stream benchmark, the model size per year is 36KB compared to the dataset size of 1.3GB, approximately 37,865:1).

  • In addition to this fundamental motivation, we would like to share further insights:

    • (Privacy Risks): In common continuous modeling tasks such as vision and text, a key improvement direction is to avoid accessing historical data, as this poses privacy risks beyond storage costs [1]. Accessing models that store knowledge from historical data is clearly safer. Common improvements, such as regularization-based and prototype-based methods, are moving in this direction.

    • (Practical Impossibility): Unlike vision and text tasks, spatio-temporal graphs have a unique property: their nodes are constantly changing. This introduces practical issues that make it nearly impossible to implement. For example, when training a neural network, data must be fixed into a certain format for each batch to fully leverage GPU batch processing capabilities. The number of nodes in spatio-temporal graphs changes across different periods, making this impractical. Therefore, most methods seek a backbone STGNN independent of node count to accept spatio-temporal graph data from different time periods, but it still requires that node counts must be consistent during training within the same period.

    • (Existing Approximation Methods): The existing Online-ST methods can be seen as an approximation solution to training with all historical data. However, this often suffers from catastrophic forgetting and the need for full parameter adjustments, issues that our EAC effectively addresses.

• We sincerely hope these insights help clear up the misunderstanding.

[1] Wang, et al. "A comprehensive survey of continual learning: theory, method and application." IEEE TPAMI, 2024.

I appreciate the authors' effort put into addressing my concerns. Regarding the prompt pool design, while Appendix provide explanations, the clarity of the integration process for new sensors in dynamic environments could be improved. I recommend revising the main text to explicitly highlight this process, potentially with the inclusion of a simplified diagram to enhance understanding.

I also suggest incorporating the results of separate period evaluations into the revised paper, perhaps in the appendix. Including a brief explanation of how noise and varying difficulty levels affect performance across periods, along with a discussion of why the proposed approach outperforms retraining for each period, would further strengthen the paper's contribution.

If these revisions are addressed, I will raise my score.

Dear Reviewer vM4w,

We sincerely thank you for your hard work on our paper. We carefully considered each of your comments and dealt with them accordingly.

---

Q1: Discussing more related works?

• We are happy to provide further discussion on these works. Specifically:
  • Unist [1]: This work can be viewed as a parallel effort in two different directions. It focuses on large-scale pretraining in the initial stage. While it also uses some empirical prompt learning methods for fine-tuning, it is not suitable for continuous incremental scenarios. Another key point is that it is limited to spatiotemporal grid data and cannot be applied to spatiotemporal graph scenarios.
  • FlashST [2]: This work is essentially limited to static spatiotemporal graphs, adjusting data distributions via prompt embedding regularization to achieve efficient model adaptation across different spatiotemporal prediction tasks. Therefore, it cannot be reasonably applied to continuous spatiotemporal graph learning.
  • CMuST [3]: This recent outstanding work addresses spatiotemporal learning in continuous multitask scenarios, enhancing individual tasks by jointly modeling learning tasks in the same spatiotemporal domain. However, this framework mainly focuses on task-level continual learning, transitioning from one task to another. It does not address the continuous spatiotemporal graph learning characteristics encountered in real spatiotemporal prediction scenarios. Hence, this approach is also not suitable for direct comparison to our single-task dynamic forecasting scenarios.
  • We hope these insights help clarify the unique aspects of our work.

[1] Yuan, Yuan, et al. "Unist: a prompt-empowered universal model for urban spatio-temporal prediction." SIGKDD, 2024.

[2] Li, Zhonghang, et al. "FlashST: A Simple and Universal Prompt-Tuning Framework for Traffic Prediction." ICML, 2024.

[3] Yi, Zhongchao, et al. "Get Rid of Task Isolation: A Continuous Multi-task Spatio-Temporal Learning Framework." NIPS, 2024.

---

Q2: The issue of parameter explosion needs to be discussed.

• We are happy to clarify further. In addition to the analyses in the paper, here are deeper insights:
  • (Adjustable Parameter Count): As we introduced in Section 4.2, there is no free lunch in this regard. We acknowledge that while introducing the prompt parameter pool significantly improves performance, there is an inevitable risk of parameter bloat as the dataset size grows with more nodes. Therefore, we dedicated an entire section to empirical observations and theoretical analysis to explain how to reasonably eliminate redundant parameters, offering a compress principle. Consequently, the adjustable parameter count can be dynamically reduced with changes in k. In the hyperparameter analysis section, we further demonstrate that even with a limited number of tuning parameters, our approach still achieves SOTA performance. Thus, in large-scale scenarios, we can appropriately select the hyperparameter k to balance performance and efficiency.
  • (Freezing Backbone Model): One of the main advantages of EAC is that, compared to existing methods, freezing the backbone model directly leads to significant efficiency improvements, even in large-scale datasets. For example, in Figure 6, we maintained the fastest average training speed on the air-stream dataset, and this can be further accelerated by adjusting parameter k. Therefore, our approach is clearly the best choice compared to others.
  • (Advantages of In-Memory Storage): Another point worth noting is that we need to point out that our prompt parameter pool is separate from the backbone model. Therefore, we can naturally store it in memory and only load it when needed. Therefore, for practical applications, overloading the prompt parameter pool is unnecessary worry.
  • (New Largest-Scale Benchmark): We also would like to point out that the current mainstream benchmark for continuous spatio-temporal learning is PEMS-Stream, which has over 800 nodes. In this paper, we further gathered and constructed benchmark datasets from various domains (including meteorology and energy) and different scales (with more and fewer nodes), aiming to provide a richer evaluation for future work. Notably, Air-Stream includes spatio-temporal data from air monitoring stations across China, which we believe is practical for deployment. As for even larger global datasets, the current backbone model size would clearly be insufficient and would need to be scaled up, as seen in current large-scale spatio-temporal models [4, 5]. In contrast, the growth of the prompt parameter pool can be considered a more manageable solution.

[4] Lam, et al. "GraphCast: Learning skillful medium-range global weather forecasting." Science,, 2023.

[5] Shi, et al. "Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts." arXiv, 2024.

---

Q3: Would EAC benefit from large-scale STGNN backbones?

Yes, although we don’t emphasize this, we do mention that the results in Table 1 are not our upper limit. For example, in Table 3, by changing the backbone, the performance of our method improves further. We believe this benefit is not captured by other methods.

---

Q4: How does EAC adapt to graph reduction, and why was this scenario not compared?

We are happy to clarify to avoid any misunderstandings. As discussed in the appendix, our method easily adapts to graph reduction scenarios. EAC uses node-level prompts, so for nodes that disappear in a new spatiotemporal graph, we simply do not load the corresponding prompt parameters. The reason this scenario wasn't compared is twofold:

• (No Suitable Real-world Datasets): Firstly, in real-world observation stations, once established, they are rarely removed, so there is almost no real-world spatio-temporal graph reduction dataset.
• (No Suitable Comparison Baselines): Secondly, current baselines cannot handle this setup, so we omitted this comparison.

---

Thank you for your valuable feedback. We will incorporate these detailed discussions into the final revised version. Once again, we appreciate your guidance!

Dear Reviewer vM4w,

Since the End of the Rebuttal is coming very soon - only a few days left, we would like to inquire if our response addresses your primary concerns. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper.

Thanks again for dedicating your time to enhancing our paper!

Looking forward to your feedback.

Best,

Authors

Dear Reviewer 7vMi,

We sincerely appreciate the time and effort you have spent providing insightful feedback on our paper. We are honored that you recognized our hard work. We have carefully considered each of your comments and have addressed them one by one.

---

Q1: Discussing more related works, and why they were not compared?

• We would be happy to clarify this point. Specifically:

  • Regarding the work combining reinforcement learning, ST-CRL [1], we did not include it because of its inaccessible code and non-reproducible methodological examples. Additionally, another consideration was its poor results. Notably, in the original paper, the comparison between ST-CRL and our method on the PEMS-Stream benchmark is as follows (Table 1):

  Horizon		3			12
  Metric	MAE	RMSE	MAPE	MAE	RMSE	MAPE
  ST-CRL	18.41	24.63	22.64	24.45	35.11	29.40
  Our	12.65±0.03	20.24±0.06	17.80±0.08	14.92±0.11	24.17±0.17	20.82±0.16

  • Regarding the work combining data augmentation, URCL [2], this work essentially treats the spatio-temporal graph as static, with only the observed instances changing over time. Therefore, this method cannot be directly compared with ours.

We hope these insights help clarify any misunderstandings.

[1] Xiao, et al. "Streaming Traffic Flow Prediction Based on Continuous Reinforcement Learning," ICDMW, 2022.

[2] Miao, et al. "A unified replay-based continuous learning framework for spatio-temporal prediction on streaming data." ICDE, 2024.

---

Q2: The issue of parameter explosion needs to be discussed.

• We are happy to provide further clarification. Beyond the analysis in the paper, here are some deeper insights:

  • (Adjustable Parameter Count): As we introduced in Section 4.2, there is no free lunch in this regard. We acknowledge that while introducing the prompt parameter pool significantly improves performance, there is an inevitable risk of parameter bloat as the dataset size grows with more nodes. Therefore, we dedicated an entire section to empirical observations and theoretical analysis to explain how to reasonably eliminate redundant parameters, offering a compress principle. Consequently, the adjustable parameter count can be dynamically reduced with changes in k. In the hyperparameter analysis section, we further demonstrate that even with a limited number of tuning parameters, our approach still achieves SOTA performance. Thus, in large-scale scenarios, we can appropriately select the hyperparameter k to balance performance and efficiency.
  • (Freezing Backbone Model): One of the main advantages of EAC is that, compared to existing methods, freezing the backbone model directly leads to significant efficiency improvements, even in large-scale datasets. For example, in Figure 6, we maintained the fastest average training speed on the air-stream dataset, and this can be further accelerated by adjusting parameter k. Therefore, our approach is clearly the best choice compared to others.
  • (Advantages of In-Memory Storage): Another point worth noting is that we need to point out that our prompt parameter pool is separate from the backbone model. Therefore, we can naturally store it in memory and only load it when needed. Therefore, for practical applications, overloading the prompt parameter pool is unnecessary worry.
  • (New Largest-Scale Benchmark): We also would like to point out that the current mainstream benchmark for continuous spatio-temporal learning is PEMS-Stream, which has over 800 nodes. In this paper, we further gathered and constructed benchmark datasets from various domains (including meteorology and energy) and different scales (with more and fewer nodes), aiming to provide a richer evaluation for future work. Notably, Air-Stream includes spatio-temporal data from air monitoring stations across China, which we believe is practical for deployment. As for even larger global datasets, the current backbone model size would clearly be insufficient and would need to be scaled up, as seen in current large-scale spatio-temporal models [3, 4]. In contrast, the growth of the prompt parameter pool can be considered a more manageable solution.

[3] Lam, et al. "GraphCast: Learning skillful medium-range global weather forecasting." Science,, 2023.

[4] Shi, et al. "Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts." arXiv, 2024.

---

Thank you for your valuable feedback. We will appropriately incorporate these detailed discussions into the final revised version of our paper. Once again, we appreciate your guidance!

Dear Reviewer 7vMi,

Since the End of the Rebuttal is coming very soon - only a few days left, we would like to inquire if our response addresses your primary concerns. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper.

Thanks again for dedicating your time to enhancing our paper!

Looking forward to your feedback.

Best,

Authors

Dear Reviewer S8oR,

We greatly appreciate your hard work during the review process. We are also pleased that you recognized several key features of our work in the Summary and Strengths section: innovation and practical value, offering a new perspective, universality (applicable to various STGNNs), acceleration of training, and reduction in the number of adjustable parameters. At the same time, we fully understand your concerns regarding the experimental section and are thankful for your detailed and insightful feedback. We believe all of these are misunderstandings or omissions of information. We will address your concerns one by one to clarify any confusion. Specifically:

---

Q1: Why not use all historical spatio-temporal data for training? Could you try including results using this approach?

• We are happy to clarify this point. Stacking all historical data together for training might seem reasonable at first glance, but it is actually impractical and reflects a misunderstanding of continuous spatio-temporal graph learning. As we pointed out in the last sentence of the first paragraph of the introduction: "Due to computational and storage costs, it is often impractical to store all data and retrain the entire STGNN model from scratch for each time period." Thus, the primary motivation behind continuous spatio-temporal graph modeling methods is:

  • (Training and Storage Costs): Storing all historical data and retraining is associated with unacceptable training and storage costs. Training costs are easy to understand, but storage costs are significant because the model is usually only a fraction of the size of the data (e.g., in the PEMS-Stream benchmark, the model size per year is 36KB compared to the dataset size of 1.3GB, approximately 37,865:1).

  • In addition to this fundamental motivation, we would like to share further insights:

    • (Privacy Risks): In common continuous modeling tasks such as vision and text, a key improvement direction is to avoid accessing historical data, as this poses privacy risks beyond storage costs [1]. Accessing models that store knowledge from historical data is clearly safer. Common improvements, such as regularization-based and prototype-based methods, are moving in this direction.

    • (Practical Impossibility): Unlike vision and text tasks, spatio-temporal graphs have a unique property: their nodes are constantly changing. This introduces practical issues that make it nearly impossible to implement. For example, when training a neural network, data must be fixed into a certain format for each batch to fully leverage GPU batch processing capabilities. The number of nodes in spatio-temporal graphs changes across different periods, making this impractical. Therefore, most methods seek a backbone STGNN independent of node count to accept spatio-temporal graph data from different time periods, but it still requires that node counts must be consistent during training within the same period.

    • (Existing Approximation Methods): The existing Online-ST methods can be seen as an approximation solution to training with all historical data. However, this often suffers from catastrophic forgetting and the need for full parameter adjustments, issues that our EAC effectively addresses.

  • We sincerely hope these insights will help clear up any misunderstandings.

[1] Wang, et al. "A comprehensive survey of continual learning: theory, method and application." IEEE TPAMI, 2024.

---

Q2: Provide more discussion on the differences in results across different domains (datasets).

• We are happy to provide further analysis. Specifically, the primary difference between the three spatio-temporal datasets lies in the underlying spatio-temporal dynamics, or spatial dependencies:
  • (Differences in Underlying Spatio-temporal Dynamics): In order, Energy-Stream involves a small wind farm with closely spaced turbines that share similar spatial patterns. PEMS-Stream represents the entire transportation system in Southern California, with moderate spatial dependencies. Air-Stream goes further, encompassing air quality records from air monitoring stations across all of China, leading to much more complex spatial dependencies. These differences are reflected in the MAE and RMSE metrics, with the performance progressively degrading as the complexity increases.
  • (Special Characteristics of Wind Farm Data): Regarding the MAPE metric, we must note that in the Energy-Stream dataset, turbines occasionally face overheating protection or shutdown for inspection, which can cause some devices to produce very small values during certain time periods. This causes large percentage errors when using MAPE.

---

Q3: Would EAC be affected by dataset size? Please provide deeper analysis of the impact of data size on model performance and the potential risks in practice.

• In addition to the analysis mentioned in the paper, we are happy to provide more detailed insights to alleviate your concerns. Specifically:
  • (Adjustable Parameter Count): As we introduced in Section 4.2, there is no free lunch in this regard. We acknowledge that while introducing the prompt parameter pool significantly improves performance, there is an inevitable risk of parameter bloat as the dataset size grows with more nodes. Therefore, we dedicated an entire section to empirical observations and theoretical analysis to explain how to reasonably eliminate redundant parameters, offering a compress principle. Consequently, the adjustable parameter count can be dynamically reduced with changes in k. In the hyperparameter analysis section, we further demonstrate that even with a limited number of tuning parameters, our approach still achieves SOTA performance. Thus, in large-scale scenarios, we can appropriately select the hyperparameter k to balance performance and efficiency.
  • (Freezing Backbone Model): One of the main advantages of EAC is that, compared to existing methods, freezing the backbone model directly leads to significant efficiency improvements, even in large-scale datasets. For example, in Figure 6, we maintained the fastest average training speed on the air-stream dataset, and this can be further accelerated by adjusting parameter k. Therefore, our approach is clearly the best choice compared to others.
  • (Advantages of In-Memory Storage): Another point worth noting is that we need to point out that our prompt parameter pool is separate from the backbone model. Therefore, we can naturally store it in memory and only load it when needed. Therefore, for practical applications, overloading the prompt parameter pool is unnecessary worry.
  • (New Largest-Scale Benchmark): We also would like to point out that the current mainstream benchmark for continuous spatio-temporal learning is PEMS-Stream, which has over 800 nodes. In this paper, we further gathered and constructed benchmark datasets from various domains (including meteorology and energy) and different scales (with more and fewer nodes), aiming to provide a richer evaluation for future work. Notably, Air-Stream includes spatio-temporal data from air monitoring stations across China, which we believe is practical for deployment. As for even larger global datasets, the current backbone model size would clearly be insufficient and would need to be scaled up, as seen in current large-scale spatio-temporal models [2, 3]. In contrast, the growth of the prompt parameter pool can be considered a more manageable solution.

[2] Lam, et al. "GraphCast: Learning skillful medium-range global weather forecasting." Science,, 2023.

[3] Shi, et al. "Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts." arXiv, 2024.

---

Q4: Missing details on baselines and datasets.

• A friendly reminder: it appears that you missed the appendix materials, which fully address your concerns regarding baseline and dataset details. However, we will briefly summarize and respond to these issues here for clarity:
  • (Baseline Details): In Section 5.1, we thoroughly discuss the specific approaches for various baselines and categorize several advanced methods. We also provide detailed descriptions, code links, and parameter settings for these methods in Appendix C.2. Notably, we also offer an anonymous repository containing all experimental results, training logs, and model weights of all baselines.
  • (Dataset Details): In Section 5.1, we present the detailed information about all datasets and experimental setups. Further, in Appendix C.1, we provide additional details on dataset construction, feature selection, and statistical information.

We believe we have provided sufficient baseline and dataset details. If you need any additional specifics, please feel free to provide more precise requests.

---

Q5: Should more recent baselines from 2023 and 2024 be considered?

• (Comprehensive Survey): We are genuinely puzzled by this comment, as we have thoroughly surveyed all relevant papers in this field and compared all comparable baselines. Specifically, three of the four advanced improvements we included in our work [4, 5, 6] are from 2023 and 2024. In the Related Work section, we summarized other works that are not directly comparable but still relevant. We have also provided detailed responses to Reviewer 7vMi and Reviewer vM4w regarding why some works are not comparable. If you have specific baselines in mind, please do let us know.
• (Fair Comparison): Another important detail is that we conducted fair comparisons for all approaches, including unified parameter settings and multiple rounds of experimentation. In particular, all experiments in this paper were repeated five times with random seeds to ensure fairness. For example, for the seven baselines across three different datasets, we performed a total of 7*(4+7+4)*5=525 experiments, which incurs substantial research costs that are often overlooked.
• (Further Performance Improvement): Lastly, although we do not emphasize it, Table 1 results are not the upper limit of our approach. For example, by changing the backbone network, we can further improve performance, as shown in Table 3. We believe this benefit is not achieved by other methods.

[4] Wang, et al. "Pattern expansion and consolidation on evolving graphs for continual traffic prediction." SIGKDD, 2023.

[5] Wang, et al. "Knowledge expansion and consolidation for continual traffic prediction with expanding graphs." IEEE TITS, 2023.

[6] Lee, et al. "Continual Traffic Forecasting via Mixture of Experts.", arXiv 2024.

---

In summary, we believe our approach offers a rich exploration and provides a new option for continuous spatio-temporal graph learning tasks. Given its lightweight, efficient, effective, and universality characteristics, it has the potential to serve as a new suitable baseline. We hope the responses above address your concerns, and if possible, we kindly request that you reconsider increasing the score. Should you have any further suggestions, we are more than happy to discuss them and make necessary improvements to the paper.

Cheers,

All authors

Dear Reviewer S8oR,

Since the End of the Rebuttal is coming very soon - only a few days left, we would like to inquire if our response addresses your primary concerns. If it does, we kindly request that you reconsider the score. If you have any additional suggestions, we are more than willing to engage in further discussions and make necessary improvements to the paper.

Thanks again for dedicating your time to enhancing our paper!

Looking forward to your feedback.

Best,

Authors