Frequency Adaptive Normalization For Non-stationary Time Series Forecasting

We appreciate all your suggestions and hope that the following answers have clarified your questions.

Q1: lacking experimental descriptions, e.g. dataset metrics Trend and Seasonality Variation.

Thank you for your valuable advice. Due to page limitations, we have briefly discussed the calculation methods for the data in the main text (line 185-187, 248). Based on your suggestion, additional details will be provided in the appendix to improve clarity and reproducibility. Furthermore, we have also included a Jupyter notebook demonstrating the calculation of selected K and other metrics, e.g. Trend/Seasonal Variation, ADF test values, available on anonymous repo. Speciffically, tread/seasonal variations are calculated as follows:

Trend Variation:, given a timeseries dataset $\mathcal{X} \in \mathbb{R}^{N\times D}$, we first chronologically split it into $\mathcal{X}^{\text{train}}$, $\mathcal{X}^{\text{val}}$, and $\mathcal{X}^{\text{test}}$, representing the training, validation, and testing datasets, respectively. The trend variations are then computed as follows:

where the subscripts indicate the dimension of mean, $| \cdot |$ denotes the absolute value operation, and $\mathcal{X}^{\text{val}, \text{test}}$ represents the concatenation of the validation and test sets. Note that, to obtain relative results across different datasets, the trend variation is normalized by dividing by the mean of the training dataset. We fetch the first dimension to be the value in main content Table 1.

Sesonal Variation:Given the inputs, $X \in \mathbb{R} ^ {N_i \times L \times D}$, where $N_i$ is the number of inputs, we first obtain the FFT results of all inputs, denoted as $Z \in \mathbb{C} ^ {N_i \times L \times D}$. Then, we calculate the variance across different inputs and normalize this variance by dividing it by the mean of each input, computed as:

where the subscripts indicate the dimension of the operation. We sum the results across all channels for the value in the main text, Table 1.

Q2: lacking the ADF test values before normalization.

Thanks for the suggestion, we will include the original values in Table 1 in the latest version. Since we perform the ADF test after normalization at the instance level and RevIN's can bot handle internal non-stationarity within the input, the values before normalization and the values after applying RevIN are too similar to distinguish in Fig 4(a), as shown in the anonymous notebook. Therefore, we only plot the RevIN values for a clearer comparison. However, we will include the original values in Table 1 following your advice.

Q3: The selection of the hyperparameter K.

Although we introduced a hyperparameter K, its selection is relatively straightforward based on the distribution in the frequency domain of the dataset (Fig 7); moreover, to avoid selecting the parameter K, we provide a heuristic formula selecting frequencies greater than 10% of the maximum amplitude (line 207) which is applied across our experiments. We further illustrate the effectiveness of this selection rule in attached PDF Fig 1.

We hope our responses have adequately addressed your concerns, and we are eager to provide further insights into our study.

I appreciate the authors' detailed responses, which address my concerns and I will raise the rating of the paper to 7.

We appreciate your constructive suggestions as they can aid in enhancing our work. We hope that responses below have addressed your concerns:

Q1: performance gap in Tab 2 and reported by SAN.

Thanks for this question. We have rechecked the results, papers, and codes to identify the reasons for the discrepancy between our results and those of SAN. The main reason is the different data splitting methods used. In results of SAN, the split ratio is 6:2:2 for the ETT dataset and 7:1:2 for the other datasets, while in our experiments, we used a split ratio of 7:2:1 for all datasets (as stated on line 185). We opted for this setup to unify the experimental setup, thereby increasing the reliability of the results.

Q2: whether FAN's capabilities fully encompass those of SAN, can FAN handle changes in statistical properties?

FAN can encompass SAN if we ignore variance as both FAN and SAN can be adjusted to achieve a constant zero mean in distribution. We have provided a theoretical analysis of FAN's effect on temporal statistical properties in Sec C.3. Based on the conclusions from the analysis in Sec C.3, FAN can handle statistical changes, as the mean will be zero and the variance variations will be largely reduced.

Q3: A more detailed comparison between FAN and previous relevant methods.

A comprehensive comparison is already provided in Sec 4.3 covering aspects such as prediction performance and showcases, stationarity after normalization, model efficiency, etc. Additional results for the synthetic dataset can also be found in Sec E.1.

To address your concern, we list the table below to summarize key metrics and give as many additional details as possible:

1. ✅: better ❌: worse ⛔: average, SAN and DishTS are two sotas in recent years.
2. The actual training iteration time should be doubled to justify the average iteration time of SAN which is trained on two-stages.

We hope this has met your expectation, and we would appreciate it if you could provide any other comparisons that might help improve our work.

Q4: why FAN consumes 0.24 million parameters, which is significant compared to FITS.

Thanks for the insightful question and for bringing up the issue of parameters. In Fig 4b, the parameters are actually consist of DLinear (0.140 M), FAN (0.109M). Why FAN consumes 109k which is large compared to FITS is as follows:

Firstly, while FITS reduces the number of parameters in the input and output layers by assuming a constant dominant frequency and using upsampling, FAN aims to model varying dominant frequencies; hence, we cannot reduce parameters by modeling only a fixed subset of dominant frequencies, as FITS does.

Secondly, FAN has tunable parameters and reducing the parameters does not significantly impact performance, but we chose a rather high performance with acceptable parameters across our paper ([64,128,128]), as in the table below:

• the experiment is conducted on ETTm2 with $H=720$ without backbone.

As in table, after reducing the parameters to 7.2k, FAN can still achieve better performance to FITS (5.5k), possiblely due to our instance-wise selection of dominant frequencies (Sec 4.5).

Q5 : Does FAN's combination of a backbone offer an advantage over methods based solely on frequency components?

In Fig 9, it can be seen that the changes in the dominant frequency are relatively small compared to the changes in the residual frequencies; furthermore, our experiments in Tab 6 show that model more parameters to predict the dominant frequency did not yield better results, which may be due to the relatively small and stable variations in the dominant frequency.Therefore, incorporating the backbone indeed provides some advantages, as we use a robust model to predict the relatively small changes in the dominant frequency and a rather complex model to predict the larger variations in the residual frequencies.

Furthermore, although we did find that even without the backbone, FAN already demonstrates good predictive capabilities by just forecasting varying dominant frequencies. This combination with the backbone indeed further improves performance and we have provided an ablation study in the main content Sec 4.5 Tab 4 to show the performance variations.

Q6: can FAN enhance frequency-based methods in a similar manner to DLinear and FEDformer?

Thanks for this question. FEDformer is itself a frequency-based method. And we specifically chose this backbone to compare FAN's performance on frequency-based method. Futhermore, we discussed the results and analysis of frequency-based methods, FEDformer/TimesNet/Koopa on line 50, line 92-98, lines 293-304, and Sec E.3. Also, we provide results of FITS and FreTS at attached PDF Tab 2.

As in the result, FAN can also enhance FITS/FreTS even they are predicting based only frequency components. This is possibly due to our instance-wise selection of dominant frequencies, treating them as the primary non-stationary component and predicting the residual frequencies separately.

We hope our explanations have sufficiently answered your questions, and we value the opportunity to explain our work further.

We are grateful for your positive feedback on our work, hope that the answers provided below have resolved your inquiries:

Q1: missing related work. e.g. DEPTS (ICLR 2022), FreTS (NIPS 2024), DERITS(IJCAI 2024).

Thank you for bringing these works to our attention! These studies indeed help us enrich the related work section. Specifically, DEPTS models periodic states as hidden states and uses an expansion module to model the relationship between cycles and future prediction steps; FreTS uses MLPs to model both the channel and temporal dimensions in the frequency domain; DERITS employs reverse transformation to map the time series to the whole spectrum through K branches FDT and uses iFDT to generate output.

We will include these works to provide a comprehensive review of frequency-focused or periodicity-focused works in the camera-ready version.

Q2: more results, e.g. PatchTST.

Thanks for the advice. Here is the result of using PatchTST as backbone where FAN can indeed enhance PatchTST on most metrics. More results can be found at attached PDF Tab 2.

Based on your advice, we will try to include these results in the latest version.

We hope our explanations have sufficiently answered your questions, and we value the opportunity to explain our work further.

Dear Reviewer 3SM3,

We are grateful for your positive evaluation of our work. We greatly appreciate the time and effort you have invested in reviewing our work. If you have any further questions, concerns, or suggestions for improvement, please feel free to reach out to us.

Authors

We thank Reviewer LAap for such a comprehensive review of our work. We hope we have addressed all your concerns as follows:

Q1: the ablation study and sensitivity analysis are provided only in the appendix and only utilizes K = 4, 8, 12, 24, which should be dataset-specific and not all dataset results have been provided.

In the main content, we do provide an ablation study in Sec 4.5 and sensitivity analysis in Sec 4.4, and K ranges from 1 to 32, containing all selected K across datasets.

Futhermore, as shown in table below, due to the large differences in spectrum distribution across datasets (Fig 7), it is hard to make cross-comparisons when using different ratios in experiments. Therefore, we chose a consistent range for K to better present the results.

However, to address your concern, we provide results on all datasets at attched PDF Fig 1, with dataset-specific K ranging from 1 to 49. This will be included in the latest version.

Q2: Reproducibility.

we have double checked our code, we are sure you can reproduce the results in our paper. To assist your review, we add some model checkpoints and training logs for you to test performance and compare training process in the anonymous repo. The docs is updated accordingly.

Q3: several places where grammar and words reduce clarity and readability.

Thank you for carefully reviewing our paper. We will recheck the grammar/wording and make appropriate changes based on your suggestions.

Q4: statements indicate overconfidence: (1) Fig 12 use "convergence speed". (2) Fig 4c where all methods' MSE/MAE are decreasing.

Thank you for bringing up these issues. (1) We acknowledge that the word "convergence speed" may lack rigor; we will update the term to "better convergence" in the latest version. (2) The conclusion derivation in Fig 4c is consistent with previous approaches, e.g. Table 5 in DishTS, Fig 4 in SAN, Fig 7 in RevIN. The mentioned issues also exist in these works. We will consider changing to relative improvement rather than absolute metric values in the latest version.

Nevertheless, we are certain that except these issues, the remaining conclusions of this paper are correctly stated based on objective results.

Q5: What are the assumptions for correlation between input dimensions?

We follow a channel independence assumption for input dimensions (mentioned in line 129).

Q6: Why use z-score and its effect on results.

As in lines 192, and in previous works, e.g. SAN, z-score can "scale the dataset to the same scale, facilitating results readibility". Furthermore, it can help model training, e.g. the loss might be too small without scaling transformation. To evaluate its effect, we conduct experiments on Traffic/ETTh1 with $H=720$ and use DLinear as backbone, the MSE results are:

Traffic:

ETTh1:

As shown in tables above, without z-score, the metrics are difficult to read/present. Futhermore, the improvements made by FAN without z-score are even more pronounced. For reviewers to reproduce the result, we update the repo docs accordingly.

Q7: How the improvements are calculated? specifically for Tab 2.

Thanks for the valuable question! The formula of MAE/MSE improvements is $f^*_{D,M} (\frac{MSE_{base}-MSE_{our}}{MSE_{base}})$ except in Tab 2 where both MSE and MAE are presented; in Tab 2, the average MSE and MAE improvement is $\frac{\sum_{D,M} (MSE_{base}+MAE_{base})-\sum_{D,M} (MSE_{our}+MAE_{our})}{\sum_{D,M} (MSE_{our}+MAE_{our})}$, since we combine both MAE/MSE results to give an uniform result. We believe that the former is a standard algorithm for calculating improvements, whereas the latter is not. For clarity, we would use the former formula across our paper in the latest version and separately list the improvements in MSE and MAE results.
The improvements in line 218 regarding Tab 2 will be updated to 9.87%/7.49%, 18.87%/14.73%, 36.91%/25.20%, 16.26%/13.24%, and 20.05%/18.79% for MSE/MAE, respectively. Note that this improvements are still substantial, which highlights the overall performance of FAN.

• *: $D$ and $M$ denote datasets and models, $f$ is some operations, e.g. max, avg across dataset/model, MAE follows the same procedure.

Q8: stated percentage improvements regarding the prediction length can not be reproduced.

Thanks for your careful review! After revising the calculation formula (Q7) for improvements in Tab 2, despite the conclusion regarding prediction length remaining valid for Informer (line 223), it cannot be generalized to all models. Therefore, we will remove this conclusion in the latest version.

Q9: the comparison on line 266.

The comparison is MSE compared to second best model SAN. The MSE improvements over SAN, increases from 0.49%(L=48) to 4.37%(L=336). We will add more detail on this to increase clarity.

Q10: Sec C4, what decreased to 8? How is this seen in Fig 10?

As shown in Fig 10, numbers on the polar axis shows the mean amplitude values of each frequency across the inputs. So, the amplitude distribution mean range decreased from 80 (SAN), 70 (DishTS), 70 (RevIN) to 8 (FAN).

We hope our responses have addressed your concerns, and we appreciate the opportunity to clarify our work.