DDN: Dual-domain Dynamic Normalization for Non-stationary Time Series Forecasting

Thanks for your valuable comments of our work, we will carefully review and revise our manuscript to correct any inappropriate expressions or symbol errors. Here are responses to your concerns and questions:

Question 1: The implementation and core idea of SlideNorm bear significant resemblance to a method proposed by an existing study [1], necessitating a comparison and clarification of its distinction.

Response 1: We are pleased to find several conceptual similarities between existing study and DDN, such as:

1. Calculating the distribution characteristics by sliding windows.

2. Employing a large sliding window to obtain long-term changes with minimal bias and eliminate local high-frequency fluctuations, and a small sliding window aims at short-term rapid changes.

The main differences between two works are as follows:

Target: DDN is a plugin module for distribution prediction, and SCNN [1] is a model for future forecasting. DDN eliminates the non-stationary factors of the input series and reconstructs them for the output series. SCNN separates non-stationary factors from intermediate representation. Thus, DDN can be better compatible with existing models and replace their reversible normalization module (RevIN) without considering intermediate features.

Experiment: To validate the above statement, we replaced the original reversible normalization module in SCNN with DDN. The training epoch will only be modified from 200 to 50 due to time constraints. The OOM is out of memory.

Distribution Calculation: SCNN directly applies sliding windows of different sizes to intermediate features to obtain various statistics for long-term low-frequency and short-term high-frequency components. DDN separates high and low-frequency components by Wavelet Transform. Then employing a large sliding window on low-frequency, obtaining long-term changes with minimal bias and eliminate local high-frequency fluctuations. A small sliding window on high-frequency component aims at short-term rapid changes.

Training strategy: SCNN sets distribution characteristics as a penalty term in the loss function. In contrast, DDN builds upon the two-stage training strategy [2, 3], which means DDN does not need to modify the original loss function of baseline. Question 2: Demonstrating that DPM surpasses some representative probabilistic forecasting models would make this study more solid and convincing. The authors should justify that the improvements achieved by their method arise from the estimation of data distribution rather than the combined power of two deep learning models.

Response 2:

1. DPM predicts distribution, not future sequence. Probabilistic time series forecasting incorporates uncertainty into prediction, allowing for a more flexible description of the future sequence distribution. However, the small scale and noise of time series datasets pose challenges for probabilistic methods to provide ideal predictive performance. Meanwhile, the long computational time limits their application as a plugin. Therefore, current DPM typically uses deterministic methods based on MLP [2, 3] rather than probabilistic methods.

2. Assessing the effectiveness of distribution prediction is challenging, as the true ground truth of data distribution is unattainable. Therefore, we designed a new experiment to evaluate and validate distribution prediction performance. To ensure DPM is used for distribution prediction rather than directly predicting future series, we first train DPM to predict mean and standard deviation with distributional loss, which is calculated from the DPM outputs and the distribution extracted from horizontal series. Then, we freeze the DPM and apply it to existing models. The experiment is as follows: |||||||||| |-|-|-|-|-|-|-|-|-| ||DLinear|DLinear|+DDN|+DDN|iTransformer|iTransformer|+DDN|+DDN| |Metric|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE| |Weather|0.245|0.298|0.223|0.274|0.263|0.292|0.219|0.273| |ELC|0.166|0.264|0.165|0.263|0.162|0.258|0.155|0.255| |Traffic|0.435|0.296|0.421|0.291|0.379|0.27|0.37|0.268|

We provide the average results for prediction lengths of {96, 192, 336, 720}. These results demonstrate that the distribution prediction capability of DPM enhances the performance of existing models. Additionally, it is worthy to note that recent works [2, 3] also combine DPM with existing models, but they often show performance degradation on Weather, Electricity, and Traffic. For instance, on the Traffic dataset with SAN, the MSE metric for DLinear and iTransformer decreased by 1.53% and 1.94%, respectively.

Question 3: Limited hyperparameter sensitivity analysis, such as sliding window size and look-back window size.

Response 3: Please refer to Reviewer i5Qt's Response 1 and 3.

[1] J. Deng et al., "Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting," in IEEE Transactions on Knowledge and Data Engineering, doi: 10.1109/TKDE.2024.3371931.

[2] Liu Z, Cheng M, Li Z, et al. Adaptive normalization for non-stationary time series forecasting: A temporal slice perspective[J]. Advances in Neural Information Processing Systems, 2024, 36.

[3] Han L, Ye H J, Zhan D C. SIN: Selective and Interpretable Normalization for Long-Term Time Series Forecasting[C]//Forty-first International Conference on Machine Learning.

Thanks for your valuable comments on our work, we will carefully review and revise our manuscript to correct any inappropriate expressions or symbol errors. Here are responses to your concerns and questions:

Question 1: What is effects of window size for low and high-frequency components?

Response 1: To address your concerns, we evaluated the impact of sliding window sizes on model performance using multiple window sizes. Similar to existing work [1], a large sliding window for the low-frequency component captures long-term changes with minimal bias and eliminates local high-frequency fluctuations, while a small sliding window for the high-frequency component targets short-term rapid changes to capture rapid variations.

Here, (7, 12) indicates a window size of 7 for the high-frequency component, and the remaining window size of 12. Additionally, we will supplement our study with more relevant experiments to further alleviate the your concerns.

Question 2: Does the work also work for other SOTA methods, like PatchTST?

Response 2: We will supplement our study with more mainstream baselines currently in use. Below are additional baselines commonly used for reversible normalization. Additionally, we will include more comprehensive baselines for PatchTST, Crossformer, and SCNN, as these are of particular interest to the reviewers. OOM means out of memory.

[1] J. Deng et al., "Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting," in IEEE Transactions on Knowledge and Data Engineering, doi: 10.1109/TKDE.2024.3371931.

Thanks for your valuable comments of our work. Here are responses to your concerns and questions:

Question 1: What is effects of the way of decomposition methods?

Response 1: As highlighted by the work [1] of reviewer G9hB, for long-term distribution changes, a larger sliding window is required to eliminate local high-frequency fluctuations and obtain an estimate of the long-term distribution characteristics with minimal bias. Conversely, for short-term distribution changes, a smaller sliding window is necessary to fully capture the complex dynamics. Our approach to decomposition is similar, utilizing Wavelet Transform to separate high-frequency and low-frequency components. The low-frequency component naturally separates out local high-frequency fluctuations, allowing us to use a larger sliding window to estimate long-term distribution characteristics. Meanwhile, the high-frequency component inherently contains rapid dynamic changes, enabling us to use a smaller sliding window to quickly capture short-term distribution variations.

Question 2: What is effects of different domains, such as time/frequency/embedding space?

Response 2: In our study, we combine time and frequency domains. The frequency domain is used to enhance the extraction of short-term high-frequency non-stationary features, while the time domain is utilized to enhance the diversity of distribution characteristics during DPM training, simultaneously avoiding interference from high-frequency noise in the extracted distribution features. Regarding the embedding space, although previous work [2] has demonstrated that linear projection can extract coarse-grained distribution characteristics from the latent space and theoretically avoid interference from the adopted frequencies while achieving higher accuracy, we did not adopt this method. The reason lies in the complex variations of distribution characteristics, which make it difficult for a simple projection operation to accurately capture fine-grained distribution features.

Question 3: What are the advantages and disadvantages of the proposed methods.

Response 3: Our method has several advantages:

1. It can be directly integrated with all current models.

2. It effectively addresses the challenges posed by non-stationary data in time series prediction.

3. It maintains SOTA performance compared to a range of current reversible normalization methods.

However, our method also has some disadvantages:

1. DPM is an MLP-based network, which adds extra parameters.

2. Precisely capturing fine-grained distribution characteristics in the embedding space using a data-driven approach remains an unresolved issue.

[1] J. Deng et al., "Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting," in IEEE Transactions on Knowledge and Data Engineering, doi: 10.1109/TKDE.2024.3371931.

[2] Fan W, Wang P, Wang D, et al. Dish-ts: a general paradigm for alleviating distribution shift in time series forecasting[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(6): 7522-7529.

Thanks for your valuable comments of our work. Here are responses to your concerns and questions:

Question 1: Lack of Hyperparameter Sensitivity Analysis, such as the length of the sliding window.

Response 1: As suggested, we conduct experiments to evaluate the impact of sliding window size of our model. In the table below, we compare results with various sliding window sizes. For example, (7,12) represents sliding window sizes of 7 and 12 for high and low-frequency components.

As table we can see that a large sliding window for low-frequency components and a small one for high-frequency components often yields better results, which aligns with the viewpoint of existing work [1] mentioned by Reviewer G9hB.

Besides, we perform sensitivity analysis on other hyperparameters, including different strides and initial wavelet bases. It is observed that our performance improves when the stride is close to a point level, and different initial wavelet bases do not significantly affect the overall results. We would add these analysis in revision.

Question 2: Insufficient Theoretical and Empirical Justification.

Response 2: Our work is based on widely adopted theoretical principle, validated by thorough empirical evaluation：

1. On the theoretical side, our method draws inspiration from rolling statistics and frequency domain analysis. Particularly, rolling statistics serve as the theoretical foundation of many time series analysis and normalization works [1, 2].

2. On the empirical side, the importance and effectiveness of each component are demonstrated by the ablation study. Specifically, in Section 4.3 we experiment with using only the frequency or the time domain branch (for convenience, the table is presented below). The results show that the combination of these two branches typically outperforms a single branch, and the frequency domain branch usually outperforms the time domain branch. ||||DLinear||||||iTransformer|||| |-|-|-|-|-|-|-|-|-|-|-|-|-| ||DDN|DDN|Frequency|Frequency|Time|Time|DDN|DDN|Frequency|Frequency|Time|Time| |Metric|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE|MSE|MAE| |Weather|0.227|0.273|0.224|0.273|0.231|0.283|0.225|0.280|0.221|0.270|0.219|0.272| |Electricity|0.162|0.260|0.161|0.259|0.155|0.255|0.152|0.252|0.161|0.259|0.152|0.252| |Traffic|0.414|0.284|0.409|0.280|0.368|0.265|0.366|0.262|0.408|0.278|0.364|0.263|

3. Each component of our method is designed to address specific challenges. First, the frequency branch addresses the ignorance of high-frequency changes. Second, the time branch aims at providing multi-scale distribution statistics and balancing the high-frequency noise that may amplified in the frequency branch. Finally, these two branches are combined in a unified framework to maximize their complementary advantages.

Question 3: Inconsistent Look-back Window Selection

Response 3: Thanks for pointing it out. Inconsistent look-back window sizes do not affect the fairness of our comparison, due to the following reasons:

1. Our work presents a plugin normalization module, and the focus is on performance improvement after its use. Therefore, maintaining the same look-back window size is sufficient. Similar practices can be found in existing works [3,4].

2. Different baselines have their best performance with different look-back window sizes, so using a unified window size cannot demonstrate their full potential. For example, CI methods tend to use larger look-back windows (e.g., DLinear and PatchTST use a 336 look-back window, while Autoformer and Fedformer use 96).

For an intuitive comparision, we also conduct experiments using the same look-back window size of 96. The results are shown in the table below, from which we see that our method still obtains better results. These further demonstrate the superiority of our method.

[1] J. Deng et al. Disentangling Structured Components: Towards Adaptive, Interpretable and Scalable Time Series Forecasting.

[2] Zivot E et al. Rolling analysis of time series

[3] Liu Z, et al. Adaptive normalization for non-stationary time series forecasting: A temporal slice perspective.

[4] Han L, et al. SIN: Selective and Interpretable Normalization for Long-Term Time Series Forecasting.

Dear Reviewer i5Qt

We greatly appreciate your initial valuable comments. We hope that you could have a quick look at our responses to your concerns. It would be highly appreciated if you could kindly update the initial rating if your questions have been addressed. We are also happy to answer any additional questions before the rebuttal ends.

Best regards

Thank you for your response, my concerns have been resolved, and considering other reviewers' concerns as well, I have raised my score to 5. Also, it's better to include some of the analysis you provided above in the paper for readers to understand the method further.