Structural Knowledge Informed Continual Multivariate Time Series Forecasting

Question 8: Explain the difference of dependency structure learning between the proposed method and GTS

We have cited the GTS paper in introduction(page 2), section 2.1 in literature review(page 3) and the beginning of our dynamic graph inference module in section 3.2 (page 4). Our dependency structure learning method has multiple important differences from GTS. Firstly, GTS learns a fixed discrete edge distribution based on Gumbel reparameterization trick from the whole training dataset, and samples static dependency structures for each time series window. However, our graph inference module learns a dynamic dependency structure based on the context of each time series window, which models the structure in a more fine-grained manner and copes with the dependencies variations within each regime. Furthermore, this leads to an important difference when doing continual learning, as we emphasized in section 3.4 (page 6), Figure 1(page 2) and the second paragraph in Appendix C (page 19), our method is able to dynamically infer faithful dependency structures for existing and current regimes without accessing the memory buffer, while GTS has to access the training data and memory buffer when inferring graphs at the testing stage, which is not realistic for practical model deployment in real-world applications. Secondly, our method considers both discrete and continuous edge variable formulations with a flexible learning component and objective, while GTS focuses on the discrete edge scenario with a specialized Gumbel design. Nevertheless, we agree with the reviewer that GTS and our method share the design choice of modeling an edge for all variable pairs instead of directly modeling the whole graph’s adjacency matrix.

Question 10: The concern regarding average forgetting in the performance evaluation

As we discussed in Appendix B.3, we emphasized the superiority of average performance over average forgetting in continual learning (page 18): “Even if we provide both metrics for performance evaluation, we need to emphasize the superiority of average performance over average forgetting in continual learning. Average performance provides a direct measure of how well a learning system is performing on a task or set of tasks. It reflects the system’s ability to retain previously learned knowledge over past regimes while adapting to new information. While average forgetting is a relevant metric in assessing the memory capabilities of a learning system, it does not provide a complete picture of the learning system’s retention abilities. The average performance takes into account both the retention of old knowledge and the acquisition of new knowledge, providing a more comprehensive evaluation of the learning system’s performance. Therefore, we use average performance as the main evaluation metric and average forgetting as an auxiliary metric to measure knowledge retention and model adaptivity.” Therefore, we are supposed to firstly look at the average performance. If the average performance is very close, then we pay attention to the average forgetting to further compare the model superiority. In the aforementioned example, autoformer-der++ does not yield satisfactory average performance over past regimes, thus it is less meaningful to directly evaluate the model solely via average forgetting.

Question 11: A typo in the sentence "given the a collection" above Eq.(1) on page 5

We sincerely appreciate the reviewer for pointing out the typo here, we corrected it in our updated manuscript.

We sincerely thank the reviewer for recognizing our contribution and the constructive comments. The answers to the concerns and questions are listed below:

Weakness 1 and Question 1: Concerns regarding the definition of regime

We sincerely appreciate the reviewer for recognizing our contributions and raising concerns in terms of the definition of regimes. In the abstract and introduction, we referred “regime” to “stage” and indicated the underlying temporal dynamics and dependency structures of multivariate time series data for each regime is different (abstract, the end of the first paragraph and the beginning of the second paragraph in introduction, page 1). We also discussed this term multiple times under the context of continual learning across the introduction. As shown in Figure 1, we defined different regimes with raw multivariate time series and the corresponding structural knowledge, which are marked and annotated using different colors. Besides, in the beginning of the methodology, we gave the mathematical formulation of our continual forecasting task with a clear notion of regime. Again, we appreciate the reviewer for raising this concern and would emphasize the definition of regime at the beginning of our final manuscript and add another real-world example for demonstration.

Weakness 2: Paper omits related work on continual learning in time series and graph domains

We explicitly discussed the methods of continual learning in time series in our related work, section 2.2, where we also explicitly mentioned methods of continual learning on graphs.

Weakness 3 and Question 9: Concerns of ablation study results with main experiment results

Main experiment results and ablation study for memory budgets are average experiment results that run by using different random seeds. The results have small differences due to randomness but similar trends are observed. We will release our code and models in the final version.

Weakness 4: Lack of comparison of model complexity or training time

We compare the number of parameters for each backbone models as following table:

The number of our models' parameters is significantly smaller than the SOTA baseline models, including Autoformer, PatchTST, TimesNet and can achieve the best performance for continual learning tasks.

Question 2 Concerns of the description of catastrophic forgetting with other domain

We appreciate the reviewer for raising this question. Yes, our description of catastrophic forgetting is appropriate. We would like to emphasize the full context of catastrophic forgetting that the reviewer refers to, in the first paragraph of page 2, is about modeling variable dependencies of MTS data for forecasting tasks (which is our research focus, and this problem is still underexplored as we discussed in the literature review, the first paragraph of section 2.2 in page 3).
We refer to the definition of catastrophic forgetting in one of the most cited paper in continual learning [1], “Continual learning poses particular challenges for artificial neural networks due to the tendency for knowledge of the previously learned task(s) (e.g., task A) to be abruptly lost as information relevant to the current task (e.g., task B) is incorporated. This phenomenon, termed catastrophic forgetting, occurs specifically when the network is trained sequentially on multiple tasks because the weights in the network that are important for task A are changed to meet the objectives of task B.”. Our description fits this definition, as the so-called “task” means regime in our case, and the “knowledge” within catastrophic forgetting involves the structural knowledge of MTS data learned by dependency modeling methods. The reviewer mentions that this is a setting in which no tasks or classes are added/incremented, which is under the context of classification tasks. In essence, our continual forecasting is a regime incremental setting in regression tasks, which is analogous to class-incremental tasks in classification tasks.
Therefore, the description demonstrates the catastrophic forgetting problem under our multivariate time series forecasting task and is appropriate compared with continual learning in other domains.

[1] Kirkpatrick, James, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114, no. 13 (2017): 3521-3526.

Weakness 5 and Question 4: The discussions of concept/temporal drift, dynamic graph learning and continual learning on traditional time series. Research positions

We appreciate the reviewer for raising this concern. Firstly, we would like to clarify that the scope of our paper is different from that of graph continual learning. Graph continual learning aims to maintain the model performance of all previous tasks on a sequence of explicit topological structures that is potentially evolving. However, in our work, we resort to modeling the dependency structure of MTS data that is coupled with the temporal dynamics, and leverage it for downstream forecasting tasks. This means that the dependency structure is not fixed and implicit within each regime. In addition to that, several terms are mentioned in this comment: concept/temporal drift, dynamic graph learning, and continual learning on traditional time series. For continual learning on traditional time series, we have discussed the continual learning methods for MTS data in the section 2.2, and we are not sure which reference the reviewer refers to here. It would be helpful to facilitate the discussion if specific literature can be pointed out. Dynamic graph learning: Next, we discuss the main differences between dynamic graph learning and our method. While both dynamic graph learning and our method deal with the evolving data, dynamic graph learning focuses on adapting to the changes in the explicit graph structure and possibly features over time. Like graph continual learning, dynamic graph learning uses the explicitly given graph structure for message passing and performs downstream tasks like node classification, link prediction, or community detection. While our proposed continual multivariate time series forecasting method needs to discover underlying dependencies structure of variables over different regimes, by capturing the temporal dynamics of MTS data. This also leads to the other key difference between dynamic graph learning and our method: the former always adapts its model to cope with the change (perform well on the new data), while the primary goal of the latter is the model's capability to continually learn without forgetting previous knowledge (i.e., addressing catastrophic forgetting). The focus of methodology can be very different due to the different learning objectives. We have similar discussion regarding the differences between online learning and our setting, in the literature review, the second paragraph of section 2.2 in page 3.
Concept drift/temporal drift: Then we would like to discuss the relationship between our research scope and concept drift/temporal drift, and explain why these methods cannot serve as baselines for comparison(as the reviewer mentioned in the other question). Concept/temporal drift of MTS represents the phenomenon where the data distribution of the target MTS changes over time in unforeseen ways. Based on this notion, it is important to emphasize the continual adaptation nature of continual learning setting regarding the concept/temporal drifts, where the catastrophic forgetting is introduced due to the shifts over multiple regimes. That being said, continual MTS forecasters need to handle a variety of regimes, not just adapt to changes in data distribution for a single one, which is to some degree more realistic and challenging. Concept/temporal drift represents a specific challenge within this broader goal. Moreover, the learning objectives between concept/temporal drift methods and continual learning methods are very different. For the concept/temporal drift methods, the objective is to adapt the forecaster to these new MTS patterns by effectively detecting, responding to, and learning from these changes. In contrast, as we emphasized before, the main focus of continual learning methods is to prevent the model from forgetting previously learned knowledge (coupled temporal dynamics and variable dependencies in our scope) when adapting to new MTS distribution. Due to the aforementioned differences in adaptation nature and learning objectives, it is very difficult to make fair experimental comparisons under the same setting. Finally, regarding the reviewer’s concern, we would like to restate the position/scope of our research compared to the fields mentioned above. We start from the continual learning perspective, explore and address the challenges of MTS forecasting problem under this setting, where we resort to variable dependency modeling and characterization aided by structural knowledge. As we discussed in section 2.2, continual learning on MTS data and especially continual forecasting is still an underexplored problem, and we proposed a solution connecting both research areas. Even if the mentioned research fields and our work all tackle the evolving data, the problem formulation, challenges and learning objectives are significantly different as we discussed above.

Question 3 and Question 7: The discussions of concept/temporal drift methods are needed.

As we discussed in our response to weakness 5 and question 4, while both concept drift methods and continual learning strategies deal with changing data environments, concept/temporal drift approaches are often more reactive and focused on adapting to changes in data distribution for a specific task. In contrast, continual learning involves a broader spectrum of adaptation, focusing on learning from new data and preventing forgetting over existing data. The continual adaptation nature of continual learning and very different objectives between both fields make it very difficult to include the concept/temporal drift methods as baselines of continual learning and make fair comparisons under the same setting.
We agree with the reviewer to some extent that the concept/temporal drift and continual learning in MTS both tackle the evolving environment, and some ideas from one field can be shared to facilitate the advancement of each field. Regarding the reviewer’s concern, we have discussed the aforementioned connections in our updated manuscript. We would like to thank the reviewer for these references of temporal drifts and will try to involve more recent methods.

Question 5 and Question 6: Include the FSNet as a backbone baseline, and MIR as a continual learning baseline.

We sincerely appreciate the reviewer’s advice. However, FSNet is designed for online forecasting, which focuses on fast adaptation to newly incoming data and uses accumulated error on the training data as evaluation protocols (there is no notion of testing data to evaluate model performance at each regime). The objective of continual learning is to maintain the learned knowledge and model performance over all seen data when adapting to new data (as we discussed in the second paragraph of section 2,2, page 3). As such, the problem formulation, learning objective and evaluation protocols are very different to the continual learning setting, making it not very meaningful for comparison. DER++ and ER are commonly used experience-replayed methods for continual learning, so it is natural for us to include them as our baselines. The MIR method is also originally designed for online continual learning setting, which uses an in-batch virtual updated model to select samples that cause the highest forgetting to perform experience replay. To enable the fair comparison, we use the aforementioned MIR criteria to select a number of samples from the fixed budget memory per regime. We report the other SOTA model results below and our proposed method consistently achieves the best performance.

Traffic-CL:

Solar-CL:

The authors have addressed my concerns well. However, there are still some issues that remain. While I understand that our time is limited, I will consider this in my decision-making process.

Questions on Traffic-CL dataset

Firstly, I appreciate the authors' clarification regarding the definition of regimes, which has greatly aided my understanding. I have a question about the Traffic-CL dataset. I have a question about how the Traffic-CL dataset was used, as shown below:

Based on the constructed PEMSD3-Stream dataset, we make the following modifications to further simulate distinct regimes in the setting of continual forecasting. For each year, we rank and select the top 100 traffic sensors with the largest node degrees, based on which we randomly select 22 sensors as a set representing a part of the major traffic.

Furthermore, the authors have sampled using 22 sensors to create the different regimes for each year. My concern arises from the choice of selecting sensor nodes based on the largest node degree. As you know, the graph structure for PeMSD3 is typically created based on the distance between sensors. In the PeMSD3-Stream dataset [1], can we assume that selecting nodes based on node degree will generate distinct regimes?

This doubt leads me to suspect that the efficacy of the method proposed by the authors for Traffic-CL may be due to non-distinctive regimes. Since the nodes with the largest node degree are sampled, the dataset is already concentrated around the central nodes or hubs of the network in all regimes. For instance, in a road network, nodes with the largest degrees are likely to be located at points with heavy traffic (e.g., intersections). This could mean that areas with relatively less traffic might be overlooked in your Traffic-CL dataset. This suggests that the traffic volume at each node in each regime is likely to be already high relative to the traffic volume of others not selected. Alternatively, because nodes with high connectivity are selected, the traffic volume of nodes is likely to be similar across regimes.

I believe this is a different issue from what is visualized in Figure 3. Could you provide information on how the traffic volume distribution varies across each regime in Traffic-CL to clarify this issue?

In other words, it is necessary to reconsider whether the Traffic-CL dataset in which changes in structural dependency do not affect temporal dependency is suitable for continuous learning using structural knowledge for MTS.

Appendix B

My concerns have been somewhat addressed in Appendix B. While the explanation in Appendix B highlights the key differences between temporal/concept drift and continual learning, it might slightly oversimplify the relationship between these two areas. As you know, the temporal and concept drift methods can be an essential part of continual learning strategies.

Additionally, the distinction between dynamic graph learning and the method proposed by the authors is still not readily apparent to the reader. I understand the authors' motivation and the differences between your method and others, but I suggest that elaborating on these differences more clearly in the introduction of the paper would be beneficial. Researchers in the dynamic graph learning and concept drift communities are likely to be particularly interested in the positioning of this paper.

Minor comment

Regarding citations in the paper, the use of \cite{} makes it difficult to distinguish between \cite{} and \citet{}. It would be more reader-friendly if the authors could change \cite{} to \citep{}.

[1] Xu Chen, Junshan Wang, and Kunqing Xie. "TrafficStream: A Streaming Traffic Flow Forecasting Framework Based on Graph Neural Networks and Continual Learning,” IJCAI, 2021. https://www.ijcai.org/proceedings/2021/0498.pdf

I maintain my score due to concern regarding the regimes in the Traffic-CL dataset.

As the authors say, in graphs where local clusters are formed, selecting the top 100 nodes with the highest node degree likely results in a high probability of choosing duplicate nodes. The selected nodes still represent structural dependency within the original networks, which calls into question the claim of different regimes.

The visualizations in the paper for Traffic-CL only show the 22 nodes for each regime. From these visualizations, I cannot discern how different sensor nodes are positioned across each regime. Additionally, referring to the statistics in the Traffic-Stream paper, the rate of expansion in the graphs is not very fast. In fact, the expansion from 2014 to 2015 involves the addition of only about 12 nodes, suggesting that these added nodes are still likely to fall within local clusters.

My continued skepticism about the “distinct regimes” claimed by the authors remains one of the most disappointing aspects of the paper. I find the methods used to form distinct regimes in Solar-CL and HAR-CL to be more convincing. In Solar-CL, different 5 states are used as distinct 5 regimes, whereas the distinct regimes in Traffic-CL seem less well-defined.

We would really appreciate if the reviewer could check our earlier responses and the experimental results in the paper carefully, rather than misinterpreting it.

“As the authors say, in graphs where local clusters are formed, selecting the top 100 nodes with the highest node degree likely results in a high probability of choosing duplicate nodes.” This comment is completely opposite to our responses and experimental results, as we said “it is not the case”.

Moreover, we need to remind the reviewer that the performance evaluation part is very strong evidence showing regime differences. Specifically, on Traffic-CL (as shown in Tables 2, 8, and 9), we have observed that _seq based methods perform significantly worse than other continual learning methods. This observation is consistent with the results (_seq) on other datasets, i.e., Solar-CL and HAR-CL (for which the reviewer has admitted the validity). This is strong evidence to demonstrate the validity of our setting and the potential distribution disparity.

We sincerely thank the reviewer for recognizing our contribution and the constructive comments. The answers to the concerns and questions are listed below:

Weakness 1: How do you define different regimes for some benchmark data? Thanks for raising this question! Two datasets are related to the standard benchmarks (PEMS and Solar), but extended by fetching more data under identical settings from the same databases. To be specific, the Traffic-CL dataset is based on an existing PEMSD3-Stream dataset [1] which uses the traffic data (Performance Measurement System in California, District 3) from the year 2011 to 2017. The standard PEMS benchmarks for general multivariate time series forecasting is a subset from this database. Based on the PEMSD3-Stream dataset, we further simulate distinct regimes represented/defined by a different portion of a temporally expanding traffic network (sensors with different geo-locations) from different years.
For Solar-CL, we use the database of NREL’s Solar Power Data for Integration Studies, which contains solar power data in the United States for the year 2006. The standard benchmark uses the Alabama subset of this database for general multivariate time series forecasting. We construct different regimes by states (spatial locations) with different average annual sunlight levels based on the statistics. The detailed description of how we generate datasets with different regimes is provided in appendix B.1 (pages 14-16). [1] Chen, Xu, Junshan Wang, and Kunqing Xie. "TrafficStream: A Streaming Traffic Flow Forecasting Framework Based on Graph Neural Networks and Continual Learning." Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21)

Weakness 2: Comparison with PatchTST baseline

We agree with the reviewer that some long-term forecasting methods are beyond the scope of our research focus. The performance of PatchTST is shown as follows. (We also provide the evaluation results for Dlinear and TimesNet). Our proposed method consistently achieves the best performance.

Traffic-CL:

Solar-CL:

HAR-CL:

Please see the next cell for experiment results for Synthetic-CL dataset

Synthetic-CL:

We sincerely thank the reviewer for recognizing our contribution and the constructive comments. The answers to the concerns and questions are listed below:

Weakness1 : Concerns about comparisons with other SOTA baselines

Thank you for the comments, this paper mainly focuses on continual learning setting of multivariate short-term forecasting and graph inference. We have compared our method with the multiple state-of-the-art graph learning backbones (that model variable dependencies for forecasting). We included results of autoformer, which currently is the third place of short-term forecasting tasks according to TSLib[1]. Regarding the reviewer’s concern, here we present more state-of-the-art model results, including PatchTST[2], Dlinear[3] and TimesNet[4] as below. As for the continual learning method, we have already included commonly used baselines for comparison. Now we also adapt and use MIR[5] for memory-replay continual learning baselines. Our proposed method consistently achieves best performance and outperform all baselines.

Traffic-CL:

Solar-CL:

HAR-CL:

Synthetic-CL:

Weakness2: The dynamic structure learning is not novel.

The existing literature either learns the structure via training parameters or representations. Our novelty lies in the role of dynamic structure learning for regime characterization, rather than an individual structure learning component. It is important that we impose the consistency regularization in the dynamic structure learning process, which aligns the learned structure with the universal and task-irrelevant structural knowledge to characterize a specific regime. The joint structure modeling based on both components leads to an capability that the existing backbones cannot achieve, i.e., during the inference stage, our model can automatically infer a consistent structure solely based on the time series data, without knowing the regime and accessing the memory buffer, as shown in Figure 1: SKI-CL: Testing (page 2) and validated in Figure 4 (page 9). Furthermore, we are the first to explore and test different structure learning scenarios regarding the formulation of edge variables (discrete or continuous) and availability of structural knowledge (completed or partial) for multivariate time series forecasting, with visualizations in Figures 5-8 in appendix (page 19-21).

Question 1: Questions about structure knowledge

Thanks for raising this question. As we discussed in the introduction (the last three lines in page 1 and the first six lines in page 2) and literature review(section 2.1 lines 10-17), the structural knowledge can be prior knowledge (e.g., physical constraints like traffic network and power grids, domain knowledge of application scenarios like Mel-frequency cepstral coefficients) or the so-called ‘learned’ knowledge if the structure is directly inferred from data by traditional statistical methods(e.g., correlation, k-NN), or learned and inferred individually (e.g., transfer entropy/granger causality).

[1]THUML. "Time-Series-Library." GitHub repository, URL: https://github.com/thuml/Time-Series-Library. Accessed on November 12, 2023

[2]Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayant Kalagnanam "A time series is worth 64 words: Long-term forecasting with transformers." in Proceedings of the International Conference on Learning Representations (2023)

[3]Zeng, Ailing, Muxi Chen, Lei Zhang, and Qiang Xu. "Are transformers effective for time series forecasting?." In Proceedings of the AAAI conference on artificial intelligence(2023)

[4]Haixu Wu and Tengge Hu and Yong Liu and Hang Zhou and Jianmin Wang and Mingsheng Long. ”TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis.” in Proceedings of the International Conference on Learning Representations (2023)

[5]Rahaf, Aljundi, and Caccia Lucas. "Online Continual Learning with Maximally Interfered Retrieval." Advances in neural information processing systems 32 (2019).

Lack of comparison of model complexity or training time

We compared the number of parameters for each backbone model in the table.

This paper can be read as an incremental work that brings the existing graph continual learning problem to the MTS domain. The discussions of concept/temporal drift, dynamic graph learning and continual learning on traditional time series are needed for the research positions of this paper.

Firstly, we argued that we have discussed continual learning on time series in our manuscript. We then referred to our manuscript and clarified that the scope of our paper is different from that of the aforementioned research topics, even if those methods all tackle evolving data. We made detailed discussions regarding the differences and restated our research position in our response. We also added the discussion in the appendix of our updated manuscript.

Compared to the definitions of catastrophic forgetting in other existing domains, this is a setting in which no tasks or classes are added/incremented. Is this description appropriate compared to definitions in other domains?

We emphasized the full context of catastrophic forgetting that the reviewer refers to, which is about modeling variable dependencies of MTS data for forecasting tasks. This is our research focus, and this problem is still underexplored as we discussed in the literature review. Then we referred to the definition of catastrophic forgetting in one of the most cited papers in continual learning, and justified the appropriateness of our description. Regarding the comment “no tasks or classes are added/incremented”, which is under the context of classification tasks, we clarified that our continual forecasting is essentially a regime incremental setting in regression tasks, which is analogous to class-incremental tasks in classification tasks.

The discussions and experiments of concept/temporal drift methods are needed.

We restated the major differences between our method and these methods (detailed discussion can be found in the previous response), and indicated that it is hard to include these methods as baselines for fair comparisons due to these differences.

Include the FSNet as a backbone baseline, and MIR as a continual learning baseline.

We indicated that FSNet is an online time series forecasting method whose problem formulation, learning objective and evaluation protocols are very different to the continual learning setting, making it not very meaningful for comparison. We then adapted the MIR method (originally designed for online continual learning) to our setting and used it as the baseline. We also reported the experiment results with more backbone models, which demonstrated that our proposed method consistently achieves the best performance. The full experiment results can be found in the appendix of our updated manuscript (table 7, page 25).

Explain the difference of dependency structure learning between the proposed method and GTS

We clarified that we have discussed the GTS paper in our manuscript. We referred to our manuscript and emphasized the key differences between our method and GTS, and an important advantage of our method over GTS (our method automatically infers faithful dependency structures for the corresponding regimes without accessing the memory buffer). We agreed with the reviewer that both methods share the design choice of modeling edge variables.

** The concern regarding average forgetting in performance evaluation **

We referred to the appendix and emphasized the superiority of average performance over average forgetting in continual learning.

A typo in the sentence "given the a collection" above Eq.(1) on page 5

We appreciated the reviewer for pointing out the typo and corrected it in our updated manuscript.

New question regarding the non-distinct regime in Traffic-CL dataset

We are delighted that we have addressed the previous concerns and we demonstrated the regime distinction by referring to and discussing the worst forecasting performance of the sequential training baseline in the experiment results. We also discussed the distinctions from the dataset statistics, and further emphasized the evolving nature of this dataset that creates distribution disparity.

Remaining concern regarding the discussion of related topics in the updated manuscript

We recalled our previous responses to emphasize the validity of what we have discussed, and thanked the reviewer for the ideas, advice and meaningful discussions.

Comment on the way of citing papers

We sincerely thanked the reviewer's suggestions and would take the advice in our final manuscript.