FilterNet: Harnessing Frequency Filters for Time Series Forecasting

We appreciate your review and the positive comments regarding our paper. We would like to respond to your comments as follows, and we hope that we address all your concerns below:

W1. The paper mainly introduces two filters. The connection and difference between the two filters is not highlighted, which makes me curious about which one is better.

A1. In the literature, compared with other model architectures, MLP-based and Transformer-based methods have achieved competitive performance. Inspired by the two paradigms, we design two corresponding types of filters. i.e., PaiFilter and TexFilter. PaiFilter offer stability and efficiency, making them suitable for tasks with static relationships. In contrast, TexFilter provides the flexibility to capture dynamic dependencies, excelling in tasks that require context-sensitive analysis. As shown in Table 1, PaiFilter performs well on small datasets (e.g., ETTh1), while TexFilter excels on large datasets (e.g., Traffic) due to the ability to model the more complex and contextual correlations present in larger datasets. This dual approach allows FilterNet to effectively handle a wide range of data types and forecasting scenarios, combining the best aspects of both paradigms to achieve superior performance.

In Appendix, we have analyzed the two filters, and introduced the connections (Frequency Filter Block in Appendix A) and differences between them (Explanations about the Design of Two Filters in Appendix B).

W2. As far as I know, there exist many frequency filters in signal processing. The motivation for these two filters is not very clear.

A2. Please refer to A2 in Response to Reviewer ufqs.

W3. In experiment part, the results of TexFilter on ETTm1, ETTm2, ETTh1 and Exchange are neither the best nor the second best. Does it show the filter cannot win in many cases of forecasting? Are there any analysis or explanation about it?

A3. Inspired by self-attention mechanisms that are highly data-dependent, dynamically computing attention scores based on the input data, TexFilter is designed to learn the filter parameters generated from the input data. This makes TexFilter particularly suitable for the scenarios that require context-sensitive analysis and an understanding of sequential or structured data. According to Table 1, TexFilter has shown competitive performance in large datasets (e.g., Traffic and Electricity) because larger datasets require more contextual structures due to their complex relationships. However, for the smaller datasets (e.g., ETT and Exchange), since the relationships are relatively simpler, using a complex design carries a risk of overfitting. For example, in Figure 10, the performance with channel-shared filters achieved better results than channel-unique filters, which indicates that a simpler architecture is more suitable for smaller dataset. In such cases, TexFilter might become overly specialized to the limited data, potentially reducing its generalizability.

We have analyzed TexFilter in Appendix B and have also given explanations about the results of Table 1 in lines 219-224.

Q1. Can the authors elaborate on the connection and difference of the two filters?

A4. Please refer to A1.

Q2. Can the authors give more details about proposed filters?

A5.Both PaiFilter and TexFilter belong to frequency filters, and the difference is the filter function $\mathcal{H}\_{filter}$. For PaiFilter, $\mathcal{H}\_{filter}$ is equal to $\mathcal{H}\_{\phi}$ where $\mathcal{H}_{\phi}$ is a random initialized learnable weight. For TexFilter, $\mathcal{H}\_{filter}$ is equal to $\mathcal{H}\_{\varphi}(\mathcal{F}(\mathbf{Z}))$ where $\mathcal{H}\_{\varphi}()$ is a data-dependent frequency filter learned from the input data. We have detailed these in Equations (1), (4), (5), and (6).

Q3. Can the authors explain the lower performance of TexFilter in some cases?

A6. Please refer to A3.

Many thanks for your positive comments and constructive suggestions. We provide a point-by-point response to your comments as follows, and we hope that we address all your concerns below:

W1. The visualization setting for Figure 1 appears to be missing in Appendix C.4, and I recommend adding some references about the spectral bias of neural networks, such as [1].

A1. Thanks for your very careful reading. For Figure 1, we generate a simple signal composed of three sine functions, each with a different frequency: low-frequency, middle-frequency, and high-frequency. Then, we conducted experiments using iTransformer and FilterNet on the same signal, reported the MSE results, and visualized the prediction values and ground-truth values. We will add these in the final version, and add some references about the spectral bias of neural network.

W2. Some typo errors

A2. Thanks, we will correct it in the final version.

Q1. In ICML 2024, some works also adopt simple architectures for time series forecasting, such as SparseTSF [2]. Could you further compare the performance of FilterNet with SparseTSF?

A3. To further compare the performance of SparseTSF with FilterNet, we conduct experiments on Electricity and Traffic datasets with the look-back window length of 96. Since SparseTSF only reported the MSE results, we also report the results of MSE, and the results are shown as follows:

Overall, FilterNet outperforms SparseTSF. The performance of SparseTSF can be validated from its paper (see Table 12 in SparseTSF). SparseTSF has a simple architecture, which limits its capability for modeling complex relationships. This results in SparseTSF achieving worse performance on larger datasets. This limitation is also validated in FilterNet's performance comparison. On the larger datesets, TexFilter has better performance than PaiFilter, while on smaller datasets, PaiFilter has better than TexFilter.

Q2. The recent literature has seen papers based on foundation models for time series forecasting. Conversely, some works are designed with simple architectures, yet achieve better performance than foundation models. Why can simple models (e.g., FilterNet) also achieve competitive performance?

A4. It is a good question. As claimed in SparseTSF, the basis for accurate long-term time series forecasting lies in the inherent periodicity and trend of the data. Since the periodicity and trend information are very simple, and they can be modeled by a simple network, such as 1k parameters in SparseTSF. In a recent paper [1], it discovers that LLM-based methods perform the same or worse than basic ablations, yet require orders of magnitude more compute. It shows that replacing language models with simple attention layers, basic transformer blocks, randomly-initialized language models, and even removing the language model entirely, yields comparable or better performance. Also, in the paper [1], it finds that simple methods (patching with one-layer attention) can work so well. Combining the findings of FilterNet, we can conclude that simple models can also achieve competitive performance.

[1] Are Language Models Actually Useful for Time Series Forecasting? https://arxiv.org/abs/2406.16964

We appreciate your review regarding our paper. We would like to respond to your comments and hope that we address all your concerns below:

W1. Empirical comparison with results from a closely related baseline, FiLM are not shown, and FiLM reports better results for all but one dataset.

A1.

1. We have compared FilterNet with the newest representative SOTA methods, i.e., the Transformer-based method iTransformer (ICLR, 2024) and the frequency-based method FITS (ICLR, 2024). Note that FITS has already compared with FiLM (NeurIPS, 2022). According to the reported results from FITS (Tables 8 and 9), it shows that FITS outperforms FiLM. Therefore, we only compare with FITS and do not choose FiLM as a baseline.
2. Note that FiLM conducted experiments with various input length (as described in the caption of Table 1 in FiLM), and the input length is tuned for best forecasting performance. In contrast, in the FilterNet, we follow the settings of iTransformer (ICLR, 2024), PatchTST (ICLR, 2023), and FEDformer (ICML, 2022), and set the input length fixed, such as 96 in Table 1. As a result, direct comparison to the findings in the original FiLM paper is not feasible. To further compare FilterNet with FiLM, we conduct additional experiments with a fixed input length of 96, and the results are shown in below:

Also, these results of FiLM with the input length of 96 can be validated by other papers, such as [1] (see its Table 13). According to the above table, we can find that overall, FilterNet outperforms FiLM. We will add these results in the final version.

[1]TIMEMIXER: DECOMPOSABLE MULTISCALE MIXING FOR TIME SERIES FORECASTING, ICLR, 2024

W2. Conceptual differences to existing layers using Fourier transforms are not discussed in detail.So far, we just learn that other architectures also use "other network architectures, such as Transformers". But a more detailed delineation would be useful, esp. as FiLM is not using Transformers.

A2. We organize the time series modeling using Fourier transforms from two perspectives: the incorporated neural network and the filter types. Autoformer and FEDformer build on the Transformer architecture and enhance its effectiveness by leveraging Fourier technologies. FreTS builds on MLP and enhance its modeling capability by leveraging frequency technologies. Although FITS and FiLM do not incorporate network architectures, they function as low-pass filters. In contrast, FilterNet is an all-pass filter to avoid information loss and also does not incorporate network architectures, ensuring high efficiency.

We will reorganize the discussion in the final version to make it clearer.

minor comments

A3. Thanks for your comments. We will correct it in the final version.

Q1. Why do you not report the better results of FiLM?

A4. Please refer to A1.

Q2. If FiLM overall is the better performing model, why is FilterNet still interesting? How is it simpler than FiLM?

A5.

1. About the performance, please refer to A1.
2. About the efficiency, please refer to A6.
3. About the architecture, FiLM first leverages a Legendre Projection Layer to obtain a compact representation and then employs a Frequency Enhance Layer to learn the projection of representations. In contrast, FilterNet uses only a Frequency Filter Block to model the temporal dependencies.

Q3. Is it maybe faster to train?

A6. Both FiLM and FilterNet conduct complex multiplication between the input and the learnable weights. However, the learnable weights in FiLM are 3-dimensional $W \in \mathbf{C}^{L\times N\times N}$ where N is the number of Legendre Polynomials, while in FilterNet they are 1-dimensional $W \in \mathbf{C}^{1\times L}$. This makes the calculation in FiLM more complicated than in FilterNet. To further compare the efficiency of FiLM with FilterNet, we conduct efficiency experiments on the Exchange dataset, and the results are shown below:

From the table, we can find that FilterNet has faster training speed and less memory usage than FiLM. The efficiency of other models is shown in Figure 8.

Dear Reviewer 2waa,

We appreciate your time and effort in reviewing our paper. As the discussion period is drawing to a close, we wanted to kindly inquire if there are any further concerns. Your feedbacks are really important to us, and we are also looking forward to further discussions with you before the discussion phase ends. We value your input and would appreciate the opportunity to respond.

Thank you again for your valuable time and effort.

Best,

Authors

Many thanks for your constructive comments and suggestions. We provide a point-by-point response to your comments below:

W1. The marginal improvement over existing approaches, as shown in Table 1, raises doubts about the compelling rationale for adopting the proposed method.

A1. The primary contribution of this paper lies in offering a novel viewpoint from signal processing and devising a simple yet effecitve architecture centered around a Frequency Filter Block. In contrast to prior research, FilterNet show remarkable efficiency (as shown in Figure 8) while simultaneously achieving state-of-the-art performance. Furthermore, we extensively investigate the efficacy and insights of frequency filters via introducing two distinct types of frequency filters, each exhibiting diversified performances across datasets of different scales. This exploration reveal the impact of frequency filters on time series forecasting and indicates that designing tailored architectures adaptive to data scale may enlighten a new path for future research endeavors

W2. The discussion on related work appears lacking. It is common knowledge that frequency filters are widely used in traditional time series analysis methods and signal processing. Therefore, a detailed exploration of the connections and distinctions between them is warranted in the related work section.

A2. In signal processing, filters are employed to selectively enhance or attenuate specific frequency components of a signal. There are various types of filters, including low-pass filter, band-pass filter, Butterworth filter, and Chebyshev filter, etc. The choice of filter type is highly dependent on the signal characteristics and the desired outcome. However, in time series forecasting scenarios, the signal characteristics and desired outcomes are unknown in advance. This uncertainty complicates the selection and design of appropriate filters, necessitating learnable or data-driven approaches to effectively capture and forecast the underlying patterns in the time series data. In FilterNet, unlike FITS and FiLM, which directly use low-pass filters, we leverage learnable frequency filters. Additionally, we design two filters that are inspired by Transformer and MLP architectures. This approach allows the model to learn the most relevant frequency components, improving forecasting performance without requiring prior knowledge of the signal characteristics.

We will add these discussions in the final version.

W3. The experiments have space to improve. Including more recent baselines for comparison, particularly deep learning methods that involve frequency processing, would add value. Additionally, a thorough investigation into the impact of hyperparameters is missing from the study.

A3.

1. We choose four categories of baselines: Frequency-based, Transformer-based, MLP-based, and TCN-based methods, including the latest SOTA methods, i.e., the Transformer-based iTransformer (ICLR, 2024) and the Frequency-based FITS (ICLR, 2024). We further compare FilterNet with SparseTSF (ICML, 2024) (please refer to A3 in Response to Reviewer UnVh) and FiLM (NeurIPS, 2022) (please refer to A1 in Response to Reviewer 2waa). Additionally, we include Koopa (NeurIPS, 2023), which is designed for addressing non-stationary issues in time series by disentangling time-invariant and time-variant dynamics through Fourier Filter, as shown below:

We choose two datasets: one smaller dataset (Exchange) and one larger dataset (Traffic), and we compare them with the look-back window length of 96. From the table, we can find that FilterNet outperforms Koopa whether the dataset is smaller or larger.

2. The bandwidth of frequency filters holds significant importance in the functionality of filters, and we have analyzed the bandwidth parameter in the experimental part (see lines 254-266). For PaiFilter, there are no hyperparameters other than a bandwidth parameter. For TexFilter, there is one hyperparameter $K$, and we conduct parameter sensitivity experiments about K as below:

Although adding layers can improve performance, considering resource costs with additional layers, we use a one-layer architecture (i.e., K=1), similar to DLinear, FreTS, and FiLM, which also use a one-layer architecture. We will add the analysis in the final version.

Dear Reviewer ufqs,

Thank you very much again for your time and efforts in reviewing our paper. We kindly remind that our discussion period is closing soon. We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks,

Authors

Thank you for addressing my questions. I still have the following concerns:

1. The authors have restated the technical contributions and efficiency achieved by the proposed method; however, the improvement in predictive performance is neither significant nor consistent across all datasets, particularly on the Traffic dataset. How many experimental runs were conducted for the reported results? From my understanding, the results of deep learning algorithms can vary across different runs. Can you guarantee that your improvements are statistically significant across all datasets? Additionally, what are the standard deviations across all runs?
2. The performance of TexFilter and PaiFilter varies significantly across different datasets. What is the underlying reason for this fluctuation? It’s difficult to determine when to use TexFilter versus PaiFilter. A more rigorous justification or empirical study is needed. Otherwise, I would prefer a model that demonstrates consistent performance across all scenarios, such as PatchTST and [1, 2].
3. Although the authors provide several results comparing FITS and iTransformer, more recent and powerful baselines, such as TimeXer [1] and Timer [2], should also be considered.
4. Why didn't the paper compare the efficiency of frequency-based methods (e.g., FITS, FilM) with the proposed method? Compared to other types of models, it is more necessary to compare it with them. Moreover, it would be beneficial to include TimeXer and Timer in the efficiency comparison as well.

Given the above concerns, I don't believe this paper is ready to publish on a top-tier venue like NeurIPS.

Reference:

[1] TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variables. arXiv 2024.

[2] Timer: Generative Pre-trained Transformers Are Large Time Series Models. ICML 2024.