The Power of Minimalism in Long Sequence Time-series Forecasting

We sincerely appreciate your detailed comments and positive ranking. We provide point-wise responses to your concerns below.

Q1 : Every univariate is now processed independently in the current convolutional network. Is there a specific reason for doing so except for the efficiency concerns? How would the performance change if we also include the feature dimension in the convolutional filter?

• Many thanks for your careful reading and suggestion. In LTSF tasks, maintaining channel independence has been observed to improve prediction performance as compared to channel mixing, as reported in prior research [2]. Our model utilizes group-wise convolution to cleverly achieve channel independence, while concurrently reducing model complexity. However, standard CNN uses the idea of channel mixing, which suffers from noise disturbance among the channels and reduces performance. We have conducted additional ablation experiments about depth-wise CNN and general CNN. The results are in Section C.2, Page 13 of the original manuscript. Channel independence helps reduce information confusion. When each channel focuses on capturing specific time series long-term patterns, the model is more capable of distinguishing and understanding these patterns without the noise that channel mixing might introduce. If we also include the feature dimension in the convolutional filter, the performance will degrade.
• Moreover, we also apply channel independence and channel dependence to the MLPs-based models respectively. The results can be found in Table 5 of the attached PDF file. Overall, the accuracy of the CI-based models was higher than the CD-based models.

Q2 : Why just one layer of convolution? How does the performance change if multiple layers are applied? This may help to capture longer periodic patterns or even help with the multi-period issue.

• We are grateful for your constructive comments. We want to emphasize that this manuscript does not only aim to showcase the contributions of SOTA results. Simple yet valuable basic units can serve as foundational building blocks, laying the groundwork for scalable and complex networks. We firmly believe that LTSF-Conv models serve as straightforward yet competitive basic units, exhibiting great potential for further expansion of complex structures. As the reviewers pointed out, based on our extended experiments, when applying complex network structures, the performance will be further improved. In the future, there is the potential for more valuable research to be explored in this new track.

Q3: Also mentioned in the paper, the effectiveness of relatively simple models such as DLinear and the convolutional network may largely depend on the nature of the current tasks. For much more complex time series with more features and periodic complexities, such simpler methods may not be as good as the transformer-based models.

• Thank you for pointing out this. When dealing with more complex datasets, increasing the depth, or introducing additional hierarchical structures to the network helps the model adapt better to the diversity of the data. For LTSF-Conv basic unit, exhibiting great potential for further expansion of complex structures. Whether based on MLPs, simpler convolution, or transformers, each model has contributed to the field of long sequence prediction. The development of these models has advanced the field of LTSF, providing diverse modeling choices for various tasks.

Q4: How do we determine the kernel size of the convolution, which should be critical for the forecasting task, especially for the cases where we don't know the periodic interval of the data streams?

• Thank you for pointing out this. In essence, finding the most suitable convolutional kernel size is a dynamic process, combining experimentation and consideration of specific data intricacies for optimal model performance. It's important to iterate and evaluate the model's performance to find the optimal kernel size for your specific forecasting task. In the absence of known periodic intervals, we usually employ a multi-scale analysis strategy, which enables the model to capture features at different time scales without relying on prior knowledge of periodicity. We can also use cross-validation to assess the performance of different kernel sizes.

Many thanks for your careful reading and suggestion. We appreciate that you found us some advantages.

Meanwhile, we would like to provide some insights into the contribution of this paper. When the DLinear model [1] was initially introduced, its simple linear structure sparked considerable controversy. However, it ultimately won the Best Paper Award, defying complex Transformer models in the LTSF field. Subsequently, more works have been dedicated to refining and advancing linear models, including architectural modifications and innovative training methodologies. We want to emphasize that our manuscript does not only aim to showcase the contributions of SOTA results. Our model structure is as simple as DLinear, while extensive experiments have consistently demonstrated that depth-wise convolutional units possess inherent advantages over linear units (such as DLinear, and RLinear) in LTSF. Linear mappings often struggle to capture intricate dependencies when handling data with multiple periods among channels. Although modeling each channel independently can alleviate this issue, it will significantly increase the computational cost. More experimental analyses are described in Section 7, Page 9. Based on our basic unit, more complex networks can be proposed to further improve the prediction performance of such tasks. In the future, there is the potential for more valuable research to be explored in this new track.

We would like to address your concerns point-by-point below:

Q1 : The paper lacks several important baseline models based on CNN architectures, such as SCINet and TimesNet, in the Conv-based model category (this is particularly severe, as all models in this category are proposed in this paper). MLP-based models lack models like N-Hits, and there is also a lack of references to the aforementioned models.

• Thank you for your comments. Due to space limitations, we compared representative SOTA models from recent years in original our manuscripts. Following the reviewer’s suggestion, we have added SCINet, TimesNet, MICN, and N-Hits baseline models. We adopt their official codes and only change the length of input sequences. The results are summarized in Table 2. In Table 2, we conducted an experiment within a broader window size range of {96, 336, 512, 720, 1600} for a fair comparison. It can be found in the attached supplementary material file. The results demonstrate that LTSF-Conv models consistently surpass all SCINet, TimesNet, MICN, and N-Hits on seven LTSF benchmarks. The symbol * denotes the experimental results we conducted after increasing the input length (re-implementation). For N-Hits, authors have adopted an extended step size hyperparameter search, and we directly adopt the results. We will add the related references in the revised paper.

Q2 : Considering Table 1 and Table 2, which compare the model results, the "best" model used in the tables is actually the model with a lookback of 1600, which naturally performs better than other baseline models with smaller lookback values that have not reached their optimal states. Moreover, even the "best" model is outperformed by PatchTST, which does not have data with lookback values of 720, 1000, and 1600.

• Thank you for pointing out this. Previous research [1] has also shown that Transformer-based baselines have not benefited from a longer look-back window. Their performance fluctuates or gets worse as the input lengths increase. We have conducted experiments to analyze it. The reason why most transformer models do not increase the lookback window length is because the performance will become worse. More analysis of the look-back windows is described in Section C.1, Page 12. in the original Appendix.
• We have confidence in the fairness of our previous comparison. The experimental results not only include Conv-Best and DConv-Best but also the default look-back window length is set at 512, such as Conv and Dconv. This is using the same lookback window length as PatchTST.
• Different from other Transformer-based baselines, PatchTST can extend the lookback window to 512, but does not benefit from longer windows. It's important to note that this extension comes at the cost of a substantial increase in the model’s computational resource requirements. On our server resources, PatchTST ran out of GPU memory for a look-back window size greater than 720.
• To alleviate your concerns, we switched to a higher-performance server to validate PatchTST with lookback-window size $\in$ {720, 1600}. Since our model only experiments from the extended window {336, 512, 720, 1600}, we ignore the 1000 step size. The results are in Table 3 of the attached supplementary material file. As expected, PatchTST did not benefit from longer lookback windows. As the input length increases, the significant memory overhead poses limitations for its practical application.

We sincerely appreciate your comments and positive ranking. In order to ensure an unbiased evaluation, we rigorously adhered to an identical code structure as that of the other baseline models. This encompassed utilizing the same data preprocessing procedures, including the data provider module. Our contribution was seamlessly integrated by adding our method to the model folder. ALL experimental results can be repeated by utilizing the provided source code. This simple basic unit serves as a robust starting point, showcasing its potential in handling LTSF tasks.

We provide point-wise responses to your concerns below.

Q1 : Whether the Conv-LTSF still works for time series that does not exhibit strong periodicity?

• Many thanks for your careful reading and suggestion. Following the reviewer’s suggestion, we compared the proposed model (Conv) with the DLinear and strongest Transformer baseline (PatchTST) on the illness benchmark. The illness benchmark describes the ratio of patients seen with illness and the number of patients from 2002 to 2021. It includes weekly data from the Centers for Disease Control and Prevention of the United States (https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html). Table 4 (in the attached PDF file) provides the corresponding forecasting results. It can be found that Conv outperforms DLinear and PatchTST for all horizons.

Table 4: The forecasting results on the ILI benchmark under the setting, where sl = 104.

Q2 : I wonder whether explicitly considering the channel dependencies can help further improve the forecasting performance.

• In time series forecasting tasks, we intuitively think that channel-mixing techniques are often a superior approach, which allows the model to effectively capture the interdependencies and interactions between different channels. In channel dependencies, information from various channels is integrated efficiently, facilitating a more comprehensive representation of the complex relationships within the data. However, previous studies [2] have found that channel independence can improve performance compared to channel mixing in LTSF.
• CI: so-called channel independence; CD: so-called channel dependencie. Following the reviewer’s suggestion, we apply channel independence and channel dependence to the different models respectively. Table 5 (in the attached PDF file) summarizes the results of different channel strategies on the Weather and ETTm1 datasets. Most of the CI-based models have a higher testing accuracy than the CD-based models. For MLP-based models, the overall accuracy of the CI-based models was approximately 1%–8% higher than the CD-based models.
• For our models, we have conducted additional ablation experiments about depth-wise CNN (channel independent) and general CNN (channel dependent). The results are in Section C.2, Page 13 of the original manuscript. It can be found that CI-based models also outperform CD-based models.

Table 5: Multivariate prediction results on two benchmarks with an input length of 336. CI denotes channel-independence, and CD represents channel-dependence.

We appreciate that you found some strengths. Before responding point-by-point, we would like to provide some insights into the contribution of our work. In its initial publication, the DLinear model [1] faced widespread controversy due to its one-layer linear model structure. Despite this initial skepticism, the model ultimately achieved success and exerted a significant influence within the LTSF community. Based on this research, lots of variant models based on MLPs have emerged. However, our model structure is as simple as DLinear, while extensive experiments have consistently demonstrated that depth-wise convolutional units possess inherent advantages over Linear units of DLinear in LTSF. We firmly believe that LTSF-Conv models serve as straightforward yet competitive basic units, exhibiting great potential for further expansion of complex network structures.

Q1 : From the accuracy perspective, two recent CNN baseline models, TimesNet and MICN [1] are missing in Table 1 and Table 2. Moreover, the hyperparameter-searching is used, which makes the comparison a little bit unfair. For example, in TimesNet and MICN, the model configurations and lookback window remain the same for most of the experiments, and in PatchTST, only two configurations are considered. Based on the current results, it is hard for me to tell whether the performance gain is from the better model configuration or the proposed structure.

• Thank you for pointing out this. We have confidence in the fairness of our previous and future comparisons. First, the experimental results not only include Conv-Best and DConv-Best but also the default look-back window length is set at 512, such as Conv and Dconv in Table 1 and Table 2 of the original manuscript. This is using the same lookback window length as PatchTST (sl=512). Then, Transformer-based baselines have not benefited from a longer look-back window. The reason why most transformer models do not increase the lookback window length is because the performance will become worse. More analysis of the look-back windows is described in Section C.1, Page 12. in the original Appendix. Reference [1] has also proved it.
• PatchTST can extend the lookback window to 512, but does not benefit from longer windows. It's important to note that this extension comes at the cost of a substantial increase in the model’s computational resource requirements. On our server resources, PatchTST ran out of GPU memory for a look-back window size greater than 720. Reference [3] also conducted similar experiments, encountering the same issue of OOM. This is also a significant limitation of the PatchTST model.
• Following the reviewer’s suggestion, we switched to a higher-performance server to validate PatchTST with lookback-window size $\in$ {720, 1600}. The results are in Table 3 of the attached supplementary material. It can be found that not only the overall performance has not improved, but the training cost is almost unacceptable.
• Finally, We have added MICN and TimesNet baseline models. We adopt their official codes and only change the length of input sequences. The results are summarized in Table 2 of the attached PDF file. In Table 2, we conducted an experiment within a broader window size range of {96, 336, 512, 720, 1600}. We consistently chose the most optimal results, thus establishing robust baselines. The results demonstrate that LTSF-Conv models consistently surpass all MICN and TimesNet on seven LTSF benchmarks.

Q2 : I would expect a more in-depth analysis of why it reaches better results than vanilla CNN.

• Thank you for pointing out this. In LTSF tasks, channel independence can enhance prediction performance compared to channel mixing. Previous research [2] has also found it. For MLP-based variants, achieving channel-independent techniques typically involves multiple independent MLP sub-models, each responsible for handling a specific channel or feature within the time series. It comes at the cost of a substantial increase in the model’s computational resource requirements. As a comparison, our model utilizes group-wise convolution to cleverly achieve channel independence, while concurrently reducing model complexity. This is because depthwise convolution exhibits higher efficiency than standard convolution. In the context of group-wise convolution, a critical consideration lies in aligning the number of channels with the variable dimension and the number of filters, as defined in the initial setup. However, standard CNN uses the idea of channel mixing, which suffers from noise interference among the channels and reduces performance. We have conducted additional ablation experiments about depth-wise CNN and general CNN. The results are in Section C.2, Page 13 of the original manuscript.