Structural Knowledge Informed Continual Learning for Multivariate Time Series Forecasting

W1: Complexity of model design.

Thank you for your feedback. The proposed design is necessary to effectively leverage the graph structure information, which is crucial for modeling variable dependencies in time series within a continual learning setting. This design utilizes the structural relationships that simpler designs cannot, addressing key challenges unique to this problem domain.

W2&W3: Novelty.

Thank you for your comment. While it is true that graph structures and memory modules are not entirely new concepts, existing graph models fail to effectively reflect the correct variable dependencies in a continual learning setting. As shown in Figure 4, this limitation can significantly deteriorate performance. Our proposed approach addresses this gap by accurately capturing and utilizing variable dependencies, which is a critical improvement. Additionally, regarding memory replay, we introduce a novel sampling scheme designed to systematically select more representative samples, further enhancing the replay mechanism. These contributions collectively demonstrate the novelty and importance of our work in advancing the field.

Q1:The OFA is published in ICML2022, the more recently model should be added as baselines.

Thank you for your comments, The OFA is published in NeurIPS 2023, which can be considered as a SOTA model.

Xue Wang Liang Sun Rong Jin Tian Zhou, Peisong Niu. One Fits All: Power general time series analysis by pretrained lm. In NeurIPS, 2023.

W1: Mixing graph structure and continuous learning in the abstract and introduction.

Thank you for your valuable review comments. We would like to emphasize that continual learning for time series with variable dependencies is a critical challenge. Our contribution lies in proposing a novel graph model to effectively leverage structural information in this continual learning setting. Additionally, we introduce a more efficient memory sample selection method for replay, aimed at mitigating the forgetting problem in continual learning.

W2: Ablation study.

Thank you for your comments. We include the ablation experiments (different predicting horizon setting) in Appendix Section F and Hyperparameter analysis in Section 4.5 and Appendix Section E.

W3: Performance compared to baselines.

Thank you for your comments. Our method achieves first or second place consistently across the datasets compared to baseline methods. This demonstrates the robustness and generalizability of our approach across diverse scenarios. Such consistent high-ranking performance highlights the effectiveness of our proposed method in addressing the challenges of continual learning for time series with variable dependencies.

Q1: Evaluation metrics.

Thank you for raising this point. The metrics we report, AP (Average Precision) and AF (Average Forgetting), are widely recognized and commonly used to evaluate a model's resistance to forgetting in continual learning settings. For clarity, we have provided detailed definitions of these metrics in Section 4.1.

Q2: Question regarding structure knowledge.

Thank you for raising this question. First, we incorporate a forecasting objective to guide the learning of the graph structure, ensuring it aligns with the continual learning setting. Second, unlike directly applying structural knowledge for message passing—an approach that, as shown in Tables 7 and 8, does not yield satisfactory results (e.g., STGCN)—we instead leverage structural knowledge to guide the model in identifying and adapting across regimes. This strategic use of structural information effectively reduces performance degradation and enhances the model's robustness in dynamic environments.

W1

(1) Analysis for Figure 1.

Thanks for your comments. We use Figure 1 to demonstrate the high-level concept of sequential training to ease the understanding of continual learning for multivariate time series forecasting, where we have provided a real-world example in lines 053-061. We have provided sufficient analyses in the experiments in the main text and appendices.

(2) Other datasets.

Other benchmarks are applicable too. As we mentioned in Appendix C.1 from line 825, the Traffic-CL dataset is built upon the PEMSD3 benchmark, which is similar to METR-LA and PEMS04. We choose the PEMSD3 as it contains sufficient traffic data from 2011 to 2017, with sensors expanding across consecutive years, which is more suitable to demonstrate the continual forecasting setting.

(3) Baselines

The full experimental results covering all baselines are presented in Table 7 & 8. As we mentioned in Appendix C.2 from line 923, our collection of baselines covers latent static graph-based forecaster (e.g., AGCRN, MTGNN), latent dynamic graph-based forecaster (e.g., ESG, StemGNN), static graph-based forecaster (STGCN), non-graph-based forecasters.

Moreover, the most recent baselines like TimesNet, OFA, and iTransformer are general-purposed time series forecaster, where the state-of-the-art performance of short-term forecasting has been validated in their papers as well. In addition to that, iTransformer also captures the variable dependencies via attention.

W2: Insights our paper.

Thanks for your comments. As we have depicted in introduction (lines 49-61) and figure 1, the challenge is about the catastrophic forgetting of learned dependency structures in multivariate time series forecasting in a sequential training learning scenario where MTS are continuously accumulated under different regimes. This challenge typically leads to performance degradation and distorted dependency structures (which are also validated in the main experiments and visualizations). To address this challenge, we leverage structural knowledge to steer the forecasting model toward identifying and adapting to different regimes, and select representative MTS samples from each regime for memory replay. As such, the obtained model can maintain accurate forecasts and infer the learned structures from the existing regimes.

The dynamically changing dependency structures means the underlying varying characteristics of multivariate time series across different regimes, especially in the context of sequential training/continual learning, where the basic assumption is the underlying dependency structures are sampled from the same distribution within one regime, and from different distributions across regimes. The notion of dynamic graph learning is about capturing the variable dependencies in a more fine-grained manner, saying an input window.

Thank you for the rebuttal, but I am not fully convinced. Additionally, I still suggest that the authors adopt stronger spatial-temporal baselines. The current spatial-temporal baselines are not state-of-the-art (SOTA) and are from 2022 or earlier. Moreover, long-term time series forecasting methods are not suitable for spatial-temporal forecasting tasks. They perform significantly worse than spatial-temporal models on such tasks, and I am very confident in this point. The authors should select more appropriate baselines to validate the effectiveness of the proposed method and proactively discuss the specific problems this method addresses as well as the limitations of the algorithm.

W1: Research topic.

We have demonstrated the importance of addressing catastrophic forgetting when time series are sequentially collected (please check figure 1 and our descriptions, and the real-world example in lines 053-061). That being said, our major contributions are about proposing a unique perspective to effectively solve continual multivariate time series forecasting.

W2: Our method differs from exisiting works .

The existing literature either learns the structure via training parameters or representations. Our novelty lies in the role of dynamic structure learning for regime characterization, rather than an individual structure learning component. It is important that we impose the consistency regularization in the dynamic structure learning process, which aligns the learned structure with the universal and task-irrelevant structural knowledge to characterize a specific regime. The joint structure modeling based on both components leads to an capability that the existing backbones cannot achieve, i.e., during the inference stage, our model can automatically infer a consistent structure solely based on the time series data, without knowing the regime and accessing the memory buffer, as shown in Figure 1: SKI-CL: Testing and validated in Figure 4 in the experiments.

W3: Analysis of representation-matching memory replay scheme .

Thanks a lot for the comments.

1. We have introduced the experience-replay methods in the experiment section from lines 354-360: A herding method randomly replays training samples, a DER++ method enforces a L2 knowledge distillation loss on the previous logits, a MIR method selects samples with highest loss for experience replay.

2. Intuitively, our representation-matching replay scheme is efficient in terms of selecting samples after the most diverse partitions, instead of the whole training set, which minimizes the search space. We will provide an efficiency analysis in our future manuscripts.

(3) We agree with the reviewer that visualizing the selected samples can be helpful to demonstrate our idea. We have provided the visualization of learned dependencies structures, continual learning performance, and a case-study with time series visualizations, and we will provide the visualization of selected samples once we update the manuscript.

W4: Inference process.

Our training and inference process follows the continual learning paradigm, where the model is sequentially trained and evaluated for each regime, and the pipeline has been demonstrated in figure 1. We thank the reviewer for the advice and will consider further elaboration.

W5: Experiments.

(1) We have included the most recent state-of-the-art time series forecasting model, iTransformer that explicitly captures the variable dependencies of time series for forecasting purposes. We will consider adding more baselines. Currently continual learning tailored for time series is still underexplored, and we’ve implemented the most well-known and effective methods that serve as strong baselines in the continual learning area.

(2) The basic intention of continual learning is maintaining performance across different tasks. We intend to demonstrate the effectiveness of our SKI-CL in terms of different continual learning scenarios regarding the time series dependencies, including the regimes representing different underlying structures. We have introduced the data splits in the training details in line 1013: The data split is 6/2/2 for training/validation/testing.

W7: Clarification of the presentation.

Due to the space limitations of the main text, we have indicated that we provided the visualizations of regimes 3 and 4 for case-study. As we have mentioned in line 513, the full case study visualizations of all regimes are provided in Appendix G. The bold and underlines results mean the best and second-best performance respectively. For equation 3, $n_K$ the number of samples for K-th mode after K-partition, which is jointly defined with K. That being said, K and $n_1, \cdots, n_K$ represent the sample partition, which are the parameters that optimize our objective.

W8: Code

Please check the supplemental materials where we have attached our code.

W1

Thank you for your comments, we will adjust it and provide a high-level explanation in our updated version.

W2

Thank you for your insightful analysis of our work. We agree that calibrated regimes are indeed critical for interpretability in domain-specific applications. In our current model design, the Coverage Maximization and Representation Matching Selection processes are unsupervised, aimed to generalize to dynamic environments. In future work, we plan to incorporate supervised signals into these components to better support regime calibration tasks, thereby further enhancing the interpretability and usability of the model in real-world scenarios.

Q1

Thank you for your insightful idea and we will continue our research in this direction.

Q2

Thank you for your suggestion and we will adjust it in our updated version.