Divide-and-Conquer Time Series Forecasting with Auto-Frequency-Correlation via Cross-Channel Attention

We thank the reviewer for their efforts and time in reviewing our paper. Also, we appreciate the reviewer's perception of our contributions. We provided detailed responses to the reviewer's comments/questions as follows.

Addressing the reviewer’s comments on weakness:

(1) Thank you again for the valuable comments. In this paper, we would like to provide a "divided-and-conquer" modeling strategy for time series forecasting studies. This strategy is the spirit of the proposed framework. That is, we argue a successful forecasting approach might be designed on top of scenarios-specialized modules to capture eventful characteristics, rather than a unified framework. Please kindly refer to the Reviewer-cikd's (the first reviewer) reply.

(2) Thank you for your question regarding our choice to use GPU memory usage as a metric in our study. Our decision is informed by related works in the field, especially in works focusing on transformer-based works [1]. In recent years, a significant challenge has been their high demand for computational memory. The core context of the Transformer is the attention calculation that involves a number of matrix multiplications and matrix storages. This is essentially relevant when processing long input sequences. That is, low memory cost allows us to use longer time series observations for improving forecasting accuracy [4].

(3) We would summarize our contributions here:

1. FeCoformer for enhancement of capturing frequency features;

2. Divide-and-conquer modeling strategy;

3. As we mentioned in (2), memory cost is one of the main focuses in time series forecasting. Existing methods address this challenge using architecturally complicated Transformer-based model designs [1, 2]. This direction often decreases the forecasting performance. Namely, recent work [4] uses a very simple linear model that can outperform all of the previous models on a variety of common benchmarks, and it challenges the usefulness of Transformer for time series forecasting. Our work does not follow the line in this direction, and we draw attention to the computation mechanism of the Transformer, i.e., the matrix issues in memory cost. We hence introduce matrix approximation methods to address this challenge without any architectural care. This attempt opens a new possibility for time series forecasting.

Addressing the reviewer’s comments on the question:

The "#channel" refers to the total number of channels in the dataset. These mainstream time series datasets consist of multiple channels, ranging in quantity from 7 to 862. We use all channels presented in each dataset without selection operation.

[1] Zhou et al., Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, 2021, AAAI

[2] Zhou et al., FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, 2022, ICML

[3] Nie et al., A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, 2023, ICLR

[4] Zeng et al., Are Transformers Effective for Time Series Forecasting? 2023, AAAI

Addressing the reviewer’s comments on the questions:

Question 1:

As we know, low-frequency oscillation represents the long-term variation or periodicity, and the frequency locality usually refers to the short-term variations, that is, high-frequency components. As we described in section 3.2, real-world time series exhibit variability across different scenarios. Sometimes, areas like banking transactions, electricity consumption, and hospital foot traffic [7, 8, 9] require a focus on short-term variations. In our ablation study, Table 4(Right, "Non-FP"), and in Figure 5 of the uploaded PDF, we compare the outcomes of processing the entire frequency domain as a whole versus segmenting it into independent patches. The results, both in terms of predictive accuracy and the visualization of frequency information, clearly show the effectiveness of locality enhancement.

Question 2:

In the multi-channel time series, some of the channels may exhibit consistent correlations within specific frequency bands(seen in Fig.4 in the new updated PDF). We would use channel-wise attention to learn and understand these correlations and key frequency components, which appear in both observation and forecasting.

Question 3:

Except for ignoring some high frequencies, in Fig.1, there are some spurious components, where many components (red bar) weigh significantly higher than the ground truth and input data. A similar result can be found in Figure 4 of our uploaded PDF. We consider the representation performance of such random or Top-K selection methods (FedFormer [3], ETSformer [4], and JTFT [10]) more dependent on the selected offers to the model. This cannot treat the frequency components equally and often needs to be more balanced with some selection.

Question 4:

Through frequency band independence, our model (i)independently normalize inputs across different frequency bands and (ii)learns features within each band by learning the consistency of different channels. This approach ensures that the model allocates equal attention to all frequency bands, rather than predominantly focusing on the low-frequency components.

Question 5:

It is true that the left column is the original DFT transformation matrix. But the right column shows the resulting DFT transformation matrics by Transformer, that is, the output of the Transformer.

Question 6:

We show some experiments with/without frequency patching in Figure 5 of the uploaded PDF. We empirically consider the patch, and independent learning does not draw any attention to any other frequency component. This prevents winner-take-all of redundancy low-frequency components. The purpose of this paper is to provide such a strategy and to begin the investigation into its properties of frequency enhancement, knowing that it will take several papers to properly investigate this line of work.

(The references can be found in part 1)

We appreciate your time and detailed review, and we also thank you for the reviewer's recognition of the strength of this paper. To address the reviewer's concerns/comments, we provide the following detailed responses.

First, we would like to address the overall goal and contribution of the paper:

From a data perspective, we observed that real-world time series exhibit variability across different scenarios. Such variations can be divided into short-term and long-term variations in time series forecasting. We argue that a successful forecasting approach might be designed on top of scenarios-specialized modules to capture eventful characteristics, rather than a unified model that solely operates at either the time domain (like Informer [1] and PatchTST [2], Crossformer [5]) or the frequency domain (FEDformer [3] and ETSformer [4]). We hence propose this divided-and-conquer modeling strategy.

From a model perspective, we would enhance the capacity to capture short-term temporal variation. We hence draw our attention to the frequency domain, since models operated on the time domain practically extract the characteristics of a combination of various frequency components. This paper proposes FreCoformer to reveal what frequencies are present in a time series scenario, and in what proportions.

Further, this investigates a new solution for tackling the common memory cost challenge. Different from existing solutions that meticulously design a lightweight model architecture, we draw our attention to the computation mechanism of the Transformer, i.e., the matrix multiplication and storage arising memory cost. We hence introduce matrix approximation methods to address this challenge without any architectural care [6].

Addressing the reviewer’s comments on the weakness:

(1) As shown in Table 4(Left) of the main body, the two modules contribute differently to the final prediction on distinctive datasets. On the Weather dataset, T-Net is the main contributor (all perform second-best results), and on the ETTh1 dataset, FreCoformer becomes more important. A similar conclusion can be found in Figures 1, and 2 of the newly uploaded PDF. These results show that our framework can automatically capture different features among different datasets. Combining both modules mainly indicates better performance.

(2) I think we keep the consistency of the original one and ours. We just deploy this self-attention on two different settings.

(3) In our implementation of the Nystromformer model [6], the selection of $m$ landmarks is determined through the process described in the following:

The process involves dividing the input sequence into equal-sized segments and then summarizing each segment to form a landmark. Specifically, for a sequence of length $n$, we calculate the stride length $l$ as the ceiling of $n/m$ and then perform a sum reduction over each segment of length $l$ in the query and key matrices. This results in $m$ landmarks that represent the original sequence.

(4) This result can be found in Fig.3(a) of the main body.

Among these comparisons, PatchTST (blue line) and ours (red line) have satisfied data fit (dodgerblue discrete points). Compared to PatchTST, our method can further capture some local fluctuations upon the periodic changes.

[1] Zhou et al., Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, 2021, AAAI

[2] Nie et al., A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, 2023, ICLR

[3] Zhou et al., FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, 2022, ICML

[4] Woo et al., ETSformer: Exponential Smoothing Transformers for Time-series Forecasting, 2022, arXiv

[5] Zhang et al., Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting, 2023, ICLR

[6] Xiong et al., Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention, 2021, AAAI

[7] Cuaresma et al., Forecasting electricity spot-prices using linear univariate time-series models, 2004, Applied Energy

[8] Thompson et al., Multifractal detrended fluctuation analysis: Practical applications to financial time series, 2016, Mathematics and Computers in Simulation

[9] Hammond et al., High-frequency sensor data capture short-term variability in Fe and Mn concentrations due to hypolimnetic oxygenation and seasonal dynamics in a drinking water reservoir, 2023, Water Research

[10] Chen et al., A Joint Time-frequency Domain Transformer for Multivariate Time Series Forecasting, 2023

We appreciate the reviewer's recognition of our contribution. We respond to the comments/concerns from the reviewer as follows.

Weakness 1: Thank you again for the valuable comments. In this paper, we would like to provide a "divided-and-conquer" modeling strategy for time series forecasting studies. It consists of the FreCoformer and the design of a linear module. Real-world time series exhibit variability across different scenarios. Therefore, using two modules, our "divide and conquer" framework is designed to capture long-term and short-term variations respectively. This strategy is the spirit of the proposed framework. That is, we argue a successful forecasting approach might be designed on top of scenarios-specialized modules to capture eventful characteristics, rather than a unified framework. Please refer to the Reviewer-CiKd's (the first reviewer) reply for more experimental evidence.

The paper [1] is architecturally somehow related to our design. However, all the operations are deployed on the time domain. This difference is the main focus of this paper, that is, models operated on the time domain extract the characteristics of a combination of various frequency components. Time series forecasting would know what frequencies are present in our signal and in what proportions. Frequency analyses can highlight eventful characteristics, such as key neural waveforms or electrical activities in paper [1].

Weakness 2: As mentioned below, although DFT projects an observation of the time domain to the frequency domain, a Fourier transform of a signal tells you the frequency composition. PatchTST [2] uses patching on the time domain and aims to learn time-time correlations for forecasting. However, it results in information loss of high frequency, as shown in Figure 1 of the main body.

On the other hand, our method focuses on capturing the important frequency feature, by learning features within each band independently. Through this, FreCoformer can capture more apparent middle-to-high-frequency features related to short-term variations in time series data. Our forecasting results outperform PatchTST, particularly on datasets with a rich spectrum of frequency features, such as ETTh1. The improvement in our predictive performance is significant.

[1] Liu et al., Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers., 2022, Advances in neural information processing systems 35.

[2] Nie et al., A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, 2023, ICLR

For the "drop_last = True" issue:

We thank the reviewer for the detailed review. This setting is for keeping the batch size consistent. When set to True, it instructs the data loader to drop the last incomplete batch if the dataset size is not divisible by the batch size. We here show the comparison of “True” and “False” settings in the Table below. As we can see, there is almost no difference.

Random seeds:

For random seed experiments, we selected two data sets, ETTh1 and Weather. The selection of random numbers is 2019, 2020, 2021, 2022, and 2023. The table records the mean value and fluctuation range of the results under various experimental settings with different seed selections:

Thank you again for the valuable comments. Indeed, we would like to provide a modeling strategy, "divided-and-conquer," for time series forecasting studies. We argue a successful forecasting approach might be designed on top of scenarios-specialized modules to capture eventful characteristics.

From a data observation, real-world time series exhibit variability across different scenarios. As we introduced in section 3.2, the main body of the paper, such variations can be divided into short-term and long-term variations. We include temporal visualization on two cases of datasets in Figure 1 of the uploaded PDF. It shows apparent differences in temporal variations between these two cases. Existing methods usually develop a unified model to capture these variations. Also, they solely operate at either the time domain (e.g., Informer [1] and PatchTST [2]) or the frequency domain (FEDformer [3] and ETSformer [4]). We observed such methods often lead to information loss, especially short-term variations, that is, high-frequency components shown in Figure 1 of the main body.

Therefore, using two modules, our "divide and conquer" framework is designed to capture long-term and short-term variations independently. Meanwhile, they are complementary to each other. As shown in Table 4(Left) of the main body, the two modules contribute differently to the final prediction on distinctive datasets. On the Weather dataset, T-Net is the main contributor (all perform second-best results), and on the ETTh1 dataset, FreCoformer becomes more important. A similar conclusion can be found in Figure 2 of the uploaded PDF. These results show that our framework can automatically capture different features among different datasets. Combining both modules mainly indicates better performance.

Random control experiments:

Thank you for your valuable suggestion. In response, we have added the experiment results in Figure 3 of the uploaded PDF. This figure illustrates the results of experimenting with 200 different combinations of parameters on two datasets: the Weather dataset, characterized by predominantly low-frequency features, and the ETTm2 dataset, notable for its high-frequency features. Practically, this paper focuses on a parameter combination that consistently demonstrates stability, rather than selecting the combination that achieves the highest accuracy.

For the "drop_last = True" issue and random control experiments, please check part 2.

Q1: Thank you for the interesting question.

We would develop two effective modules that could extract different features while keeping their structures as simple as possible.

In the temporal domain, MLPs are effective due to their ability to summarize long-term temporal dependencies. The full connection computation allows MLPs to consider all input features simultaneously, extracting global semantic information of input data. It is shown in the recent paper [5] that a very simple MLP model can outperform all of the previous models on a variety of common benchmarks.

In contrast, the frequency domain presents a set of challenges. We consider Transformer could enhance the locality (patching) and learn the correlation of these local features (attention). In our design, the frequency domain module benefits from this ability, by identifying important frequency features across channels.

Q2: Yes, it corresponds to both. The two functions in the code both refer to local differences. In the paper, 'global' and 'local' refer to the 2-layer MLP architecture. In our code, we used 2 inputs for local information extraction: "local" differences and "trend" differences. Both are aimed at capturing local dynamics within the time domain. Also, the first-order differences mentioned in section 3.2 primarily refer to these local aspects, encompassing variations both within and between patches.

[1] Zhou et al., Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, 2021, AAAI

[2] Nie et al., A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, 2023, ICLR

[3] Zhou et al., FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting, 2022, ICML

[4] Woo et al., ETSformer: Exponential Smoothing Transformers for Time-series Forecasting, 2022, arXiv

[5] Zeng et al., Are Transformers Effective for Time Series Forecasting? 2023, AAAI