Evolving Multi-Scale Normalization for Time Series Forecasting under Distribution Shifts

We are grateful for your thorough examination of our manuscript and the opportunity to clarify and enhance our work based on your insightful comments. We would like to address your concerns as follows.

• Q1:

Sorry for the unclear description. In the online setting, we set the test_batchsize=1, thus the model is updated for each test sample. We have made it clearer in the manuscript.

• Q2:

We have already demonstrated the effectiveness of the proposed method on the largest traffic dataset [1] in Table 6 (where online forecasting in the year 2019 is conducted). We would like to strengthen this statement further by reconducting the experiment on this dataset and spanning a longer period (5 years, from 2017 to 2021), where results again verify the robustness of the proposed method.

[1] Largest: A benchmark dataset for large-scale traffic forecasting

• Q3:

We agree that comparing with a wider range of baselines would strengthen our experimental results. We apply the proposed method to backbone models including PatchMixer, TimeMixer, iTransformer, Koopa, and TSMixer (as the Crossgcn is not designed for time series forecasting, we adopt TSMixer instead) for online forecasting. The results are shown in the following table (The average performance for different prediction lengths {96,192,336} is reported). The results demonstrate the broad effectiveness of the proposed methods, detailed results are reported in Table 1 in the uploaded supplementary material.

• Q4:

We believe integrating multiscale information for normalization and online learning represents a key contribution. Even though SAN adopts a similar slice-based normalization from work, it relies on single-scale modeling and also neglects the distribution shift problem over time. Our method enhances it by multi-scale modeling with adaptive ensembling and alternative online learning strategy, where both parts are validated to be superior to SAN. Taking Autoformer as the backbone forecasting model, for offline forecasting, Table 14 shows the EvoMSN achieves an average improvement (overall all datasets and prediction lengths) of 6.23% in terms of MSE compared to SAN, which demonstrates the strength of the proposed multiscale approach. For online forecasting, Table 11 shows the EvoMSN achieves a more prominent average improvement of 26.33% in terms of MSE compared to SAN, which indicates modeling distribution dynamics with a multiscale view is superior and more robust than a single view approach that adopted by SAN for addressing the distribution shifts problem in a data stream. We reckon these improvements compared to SAN are non-trivial.

Even though there are some existing studies investigating multi-scale modeling, such as TimeMixer and TimesNet, our design focuses on a totally different emphasis. Compared to these methods, our proposed multi-scale modeling does not directly let the backbone forecasting model handle the raw time series. Instead, we depend on the multi-periodicity information to explicitly model the dynamics of data distribution (statistics information includes mean and standard deviations), which is the first study to do so to the best of our knowledge. Different from existing studies that design specific model architecture, our design enhances the performance of arbitrary backbone forecasting models, and the effectiveness is also shown when it is applied to TimeMixer and TimesNet.

We have carefully read the recommended reference [1] and we find the only relation to our study is the form of bi-level optimization formulation. However, their focus is on general bi-level optimization instead of specific time series forecasting problems, and the methodology has nothing in common with that of ours.

[1] On differentiating parameterized argmin and argmax problems with application to bi-level optimization.

• Weakness 1:

We conclude potential reasons for method failure as follows: 1) In the online forecasting setting, the multi-scale statistics prediction module is updated only once for each incoming new data, which may challenge the module in capturing some fast-changing statistics and will result in bad performance. Besides, the global dominant periodicity is determined according to the training data, where the relatively small training data split ratio in the online setting may cause an inappropriate periodicity extraction and thus affect the effectiveness of multi-scale slice-based analysis. 2) In the offline forecasting setting, the multi-scale statistics prediction module is only updated on the training data, where the distinct statistics evolving dynamics of training data and testing data may cause a worse performance in the evaluation stage. Despite the potential shortcomings identified above, it is essential to emphasize the overall strengths of the proposed method.

• Weakness 2:

To analyze the computational complexity, we have documented the computation time of each part throughout the whole procedure in Table 8 and Table 9. The proposed method will cause extra computational complexity and running time from two perspectives. First, the proposed method requires training the multi-scale statistics prediction module, where the complexity is the number of considered periodicities times the complexity of individual prediction modules for a single periodicity. The computational complexity of this part is independent of the complexity of the forecasting backbone model and we have designed this part to be lightweight with a two-layer MLP to avoid a large computation burden. Second, the backbone forecasting model will handle the series from multi-scale perspectives, where the complexity is the number of considered periodicities times the complexity of the backbone forecasting model. This part becomes the main computation burden when the backbone model is much more complex than the statistics prediction module. There are two possible approaches to accelerate the computation process, one is to let the multi-scale analysis in parallel (our experiment is conducted in series); another approach is to consider an attentive updation strategy in the online learning process, which is to update model only when severe distribution shift is detected instead of updating for every sample. Overally speaking, the computational complexity of our proposed method is acceptable in many real-world applications, especially for those low-frequency forecasting scenarios, such as day-ahead electricity load forecasting.

To evaluate the scalability of the proposed method, we have already demonstrated the effectiveness of the proposed method on the largest traffic dataset [1] in Table 6 (where online forecasting in the year 2019 is conducted). We would like to strengthen this statement further by reconducting the experiment on this dataset and spanning a longer period (5 years, from 2017 to 2021), where results again verify the robustness of the proposed method.

For your concern about training data, the proposed offline-online optimization strategy not only uses samples from the training data but also uses continuous data streams from the testing data to update models, where the effectiveness is validated on datasets that all from the real world. To achieve accurate multiscale statistics predictions, we have sliced the lookback and horizon windows into slices of different lengths and calculated slice statistics, where the slice statistics in the lookback windows serve as inputs to predict slice statistics in the horizon windows.

[1] Largest: A benchmark dataset for large-scale traffic forecasting

• Weakness 3:

Thanks for your constructive suggestions. We have plotted the prediction of statistics from a multi-scale view on the ETTh2 dataset in the uploaded supplementary material Fig.1, which shows the prediction module can capture the overall trend of the mean and standard deviation. We find the standard deviation is more volatile and more difficult to predict than the mean, and our future work will investigate more advanced methods to better predict these statistics.

Thank you for your detailed reply. I would like to maintain my score.

Thank you for your acknowledgment and valuable feedback on our work, we would like to address your concerns as follows.

• Weakness:

Thanks for re-stating the limitation of the proposed method that we have analyzed in Appendix B. We will further work on these perspectives to improve the technique in our future work.

• Question 1:

Thank you for your insightful comments and for drawing attention to the architecture of our multi-scale statistics prediction module. In response to your question, we have indeed considered various architectures for our prediction module. Initially, we experimented with different neural network configurations, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs), to assess their effectiveness in capturing time-series dependencies and patterns. However, after a series of experiments and cross-validations, we found that the two-layer perceptron network (MLP) provided a good balance between model complexity and predictive accuracy for our specific dataset and task. The MLP's simplicity allowed for faster training times and easier interpretation of the model's weights, which were crucial factors for our research. We provide visualization of prediction of statistics from a multi-scale view on the ETTh2 dataset in the uploaded supplementary material Fig.1, which shows the proposed prediction module can capture the overall trend of the mean and standard deviation. We find the standard deviation is more volatile and more difficult to predict than the mean, and our future work will investigate more advanced methods to better predict these statistics.

• Questions 2 & 3:

We appreciate your concern regarding the potential limitations of FFT for time series analysis. The main reason for us to apply FFT is its efficiency in extracting multi-periodicity information from time series, which is also adopted by various mainstream studies [1,2]. We mainly utilize FFT from a global perspective to determine the dominant periodicity to guide the window slicing. Our design considers ensembling multiple outputs according to their local periodicity amplitude, which can mitigate the adverse effects of inappropriate periodicity extraction and thus make our forecasting more robust. We will investigate more adaptive window slicing (such as methods based on changepoint detection) and multi-scale modeling strategies in our future work.

[1] Wu H, Hu T, Liu Y, et al. Timesnet: Temporal 2d-variation modeling for general time series analysis[J]. arXiv preprint arXiv:2210.02186, 2022.

[2] Zhou T, Ma Z, Wen Q, et al. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting[C], International conference on machine learning. PMLR, 2022: 27268-27286.

We greatly appreciate your acknowledgment of our proposal. We would like to address your concerns as follows.

• Weakness 1:

Denote the distribution of a lookback window and a horizon window at timestep $t$ as $P_t(X)$ and $P_t(Y)$, respectively. The conditional distribution that reflects the input-output mapping relationship is denoted as $P_t(Y|X)$. First, the shifts in marginal distribution happen between the lookback and horizon window $P_t(X) \neq P_t(Y)$ and also between the window of a different timestep $P_t(X) \neq P_{t_n}(X), P_t(Y) \neq P_{t_n}(Y)$. Second, the conditional distribution shifts all the time $P_t(Y|X) \neq P_{t_n}(Y|X)$, especially between the training and testing data $P_{train}(Y|X) \neq P_{test}(Y|X)$. These distribution shifts are coupled and happen simultaneously.

• Weakness 2:

The normalization approach will remove the dynamic distribution information to let the backbone forecasting model better learn from the data with a stable distribution, while the denormalization process restores the distribution information to the outputs. The key to addressing distribution shifts is to accurately model how the distribution is shifting in order to facilitate time series forecasting. The empirical justifications are shown in Table 1, Fig.6, and Table 13, which show that models' performance will be degraded by distribution shifts without the proposed adaptive normalization approach.

• Weakness 3:

We have documented the computation time of each part throughout the whole procedure in Table 8 and Table 9. The proposed method will cause extra computational complexity and running time from two perspectives. First, the proposed method requires training the multi-scale statistics prediction module, where the complexity is the number of considered periodicities times the complexity of individual prediction modules for a single periodicity. The computational complexity of this part is independent of the complexity of the forecasting backbone model and we have designed this part to be lightweight with a two-layer MLP to avoid a large computation burden. Second, the backbone forecasting model will handle the series from multi-scale perspectives, where the complexity is the number of considered periodicities times the complexity of the backbone forecasting model. This part becomes the main computation burden when the backbone model is much more complex than the statistics prediction module. There are two possible approaches to accelerate the computation process, one is to let the multi-scale analysis in parallel (our experiment is conducted in series); another approach is to consider an attentive updation strategy in the online learning process, which is to update model only when severe distribution shift is detected instead of updating for every sample. Overally speaking, the computational complexity of our proposed method is acceptable in many real-world applications, especially for those low-frequency forecasting scenarios, such as day-ahead electricity load forecasting.

• Weakness 4:

We design this specific offline-online optimization strategy for the following motivations: The offline two-stage training process facilitates the statistics prediction model to focus on capturing distribution information and the backbone model to focus on learning from stationarized data. The offline stage enables models to learn from the past and provides a generalizable starting point for subsequent online learning. The online stage helps models track evolving distributions by learning from the continuous data stream. Specifically, the alternative updation strategy aims to optimize the lower and upper objectives sequentially and avoids over-fitting problems caused by blending the training of all components. The potential limitation of the strategy is that it may encounter the plastic-elastic dilemma and forgetting problem during the online learning stage, which will be further investigated in our future work.

• Weakness 5:

We conclude potential reasons for method failure as follows: 1) In the online forecasting setting, the multi-scale statistics prediction module is updated only once for each incoming new data, which may challenge the module in capturing some fast-changing statistics and will result in bad performance. Besides, the global dominant periodicity is determined according to the training data, where the relatively small training data split ratio in the online setting may cause an inappropriate periodicity extraction and thus affect the effectiveness of multi-scale slice-based analysis. 2) In the offline forecasting setting, the multi-scale statistics prediction module is only updated on the training data, where the distinct statistics evolving dynamics of training data and testing data may cause a worse performance in the evaluation stage. Despite the potential shortcomings identified above, it is essential to emphasize the overall strengths of the proposed method.