SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusion

Thank you for your positive assessment of our work and your thoughtful feedback. We are pleased to know that you found our approach and presentation to be excellent and that you see the value of our contributions to the field. We have carefully considered your comments and here are our responses.

Additional experiments evaluating the robustness of SOFTS under varying conditions of distribution drift could provide deeper insight into its reliability.

Thanks for your advice. We have tested the robustness of SOFTS on real-world datasets and synthetic data with manually injected noise in Figure 6. More rigorous experiments need further design and we are working on it. Thanks again for your valuable advice.

Has the author currently tried any other methods for aggregation and redistribution

To keep our method neat, we use the simplest redistribution by concatenation and MLP fusion. We also experiment with 4 simple variants of aggregation, including mean pooling, max pooling, stochastic pooling, and weighted average as shown in Table 3. This simple structure is found to perform well. We agree that there would be other better designs for aggregation and redistribution. And we leave it for future exploration.

SOFTS is a linear based model, so how to handle non-linear time series prediction?

The encoding part of SOFTS (using STAR) is composed of multiple MLP layers. So it can handle non-linear time series prediction.

Thank you for your positive feedback and constructive comments on our paper. We appreciate your recognition of our work's strengths and have carefully considered your suggestions for improvement. Below, we provide detailed responses to each of your points.

The paper lacks significant innovation. The stochastic pooling itself is not novel in deep neural networks. A somewhat novel facet of the proposed model is to use the pooling to extract global representations. However, the idea of aggregate-and-dispatch the interactions between variables has already been studied in TimeXer [1].

Thanks very much for reminding us of this paper. We understand your concerns regarding the novelty of our method. However, our approach differs significantly from TimeXer. Most importantly, TimeXer still uses attention (self-attention, cross-attention) as the module for extracting correlations between tokens. In contrast, the core innovation of our paper, the STAR module (as emphasized in line 10 of the abstract, line 63 of the introduction, and line 126 of Section 3.2), is distinct from attention. This difference is detailed in Section 3.2 and illustrated in Figure 2. Additionally, TimeXer aggregates the information of series and patch embeddings using attention, which is an aggregation in the time dimension. This method does not address the problem our paper focuses on, which is how to efficiently and robustly model channel correlation. Finally, TimeXer investigates the scenario of forecasting with exogenous variables, which differs from our study of multivariate forecasting. Their proposed method is also tailored for that scenario. Therefore, in terms of application scenarios, problems to solve, and methodology, there is little overlap between our paper and TimeXer. We believe that our work offers sufficient innovation compared to this paper.

In conclusion, we sincerely thank you for mentioning this paper and offering your suggestions. To ensure the completeness of our paper, we will add a discussion on the scenarios and methods of TimeXer in the related work and future work sections. The preliminary content is as follows:

Related Work: TimeXer [1] uses self-attention to aggregate the information of series and patches and employs cross-attention to achieve interaction between endogenous and exogenous variables. It extends iTransformer to the forecasting with exogenous variables scenario. Unlike these two methods, our paper proposes an efficient channel interaction module based on MLP, achieving better performance with lower complexity.

Future Work: Future work includes exploring how to apply SOFTS and STAR to more forecasting scenarios, such as forecasting with exogenous variables [1] and forecasting with future variates.

[1] Wang, Y., Wu, H., Dong, J., Liu, Y., Qiu, Y., Zhang, H., ... & Long, M. (2024). Timexer: Empowering transformers for time series forecasting with exogenous variables. arXiv preprint arXiv:2402.19072.

It seems that all series share a core representation and MLP layers, but different variables will require different aspects of information. How can the model address these differences?

The MLP module can fuse different parts of the core representation according to the series due to its universal approximation ability. That is $MLP(s,o)$ can approximate any forecasting function if $s,o$ contains enough information. By Kolmogorov-Arnold representation theorem and DeepSets (line 146 and Section B.2). $o$ with equation (3) can approximate the function on all the series. Therefore, even if different variables will require different aspects of information, STAR can also approximate the required function.

Please increase the look-back window length of the input series to 512 and 720, and compare the performance with other baseline models. Please provide more results on abnormal channels, i.e. ETT, ECL, and Traffic dataset.

Thanks for your advice. We have added the lookback window and the abnormal channel results (on ECL, Traffic, PEMS) in the global rebuttal PDF. The ETT dataset has only 7 channels, and the embedding correlation of the channels is not very clear, so we replaced it with PEMS. And we will make corresponding modifications to the revised version.

Thank you for taking the time to review our paper and for your constructive feedback. We appreciate your recognition of the importance of time series forecasting and your acknowledgment of our work's potential impact. We have carefully considered your comments and suggestions and have made revisions to the manuscript to address your concerns. Here are our responses.

Comparison and selection among PatchTST, Transformer, and SOFTS.

We appreciate your suggestion to provide a more in-depth analysis and discussion of our results. The main difference between the three methods is in the following table.

• Embedding: The way of creating token embeddings. "Patch" means tokens are embedded by small patches (or windows) containing values of consecutive time. "Series" means embedding the whole series.

• Temporal: The way of extracting the correlation of different time steps.

• Channel: The way of extracting the correlation of different channels.

PatchTST vs SOFTS

The main difference between PatchTST and SOFTS lies in their approach to handling channels. PatchTST uses a channel-independent model, which sacrifices channel correlation but eliminates interference from other channels in the prediction process. In contrast, our SOFTS model employs the STAR module to extract channel correlations more robustly and effectively while reducing interference from other channels. By adjusting the representation through channel clustering, as shown in Figure 6, the predictions become more robust, making SOFTS particularly advantageous when there are many channels. Although SOFTS minimizes the negative impact of channel correlation as much as possible, there is still a risk of overfitting in situations where channels are highly independent. Current multivariate datasets may not be so independent so we find SOFTS outperform in most cases.

The second difference is the patch embedding of PatchTST, which utilizes a sliding window to extract information within consecutive times. PatchTST then exchanges the information at different time windows by attention. Our SOFTS uses MLP which is simpler but achieves SOTA performance as well.

iTransformer vs SOFTS

The main difference between iTransformer and SOFTS is that iTransformer uses attention for channel interaction, while SOFTS uses STAR. As shown in Figure 2, the significant difference is that STAR employs a centralized structure to enhance robustness, avoiding the influence of certain abnormal channels on the predictions of other channels. From Table 2, it is evident that SOFTS has a clear advantage over iTransformer on datasets with a large number of channels, such as Traffic and PEMS. This is likely because, with more channels, attention is more susceptible to the influence of certain abnormal channels, whereas STAR mitigates this effect through aggregation. Additionally, in terms of efficiency, STAR reduces complexity from quadratic to linear, significantly enhancing its scalability. But we also note that SOFTS may have a bottleneck due to interaction through core representation.

So, under what conditions should I choose SOFTS over iTransformer? Based on the current results, we find that SOFTS generally outperforms iTransformer in time series problems, both in terms of performance and efficiency, especially when the number of channels is large. This might be because robustness considerations outweigh capacity in time series issues. Of course, we do not rule out the possibility of scenarios where the reverse might be true in the future.

Limitation of SOFTS

We acknowledge your suggestion to move the discussion of limitations from the appendix to the main text. In response, we have integrated Section G from the appendix into the main body of the paper. We have also enhanced this section by comparing and contrasting the limitations of SOFTS with those of other models, such as PatchTST and iTransformer, like the discussions stated above.

Once again, we sincerely thank you for your valuable feedback. Your insights have significantly contributed to improving the clarity and impact of our paper. We hope the revised manuscript addresses your concerns and enhances the overall quality of our work.

Thank you for your thorough review of our paper. We appreciate your feedback and have made corresponding responses to address your concerns.

It is unclear why different datasets and metrics are used for different ablation studies, e.g., the datasets used in Table 3 and 4 are different, MAE is used in Figure 4 but MSE is used in other figures.

Datasets of Table 3,4 are selected due to space restriction. We provide full results in the global rebuttal part. We found that MSE lines of SOTA methods usually entangle with each other, therefore we display MAE. To alleviate concerns, we have also included performance curves using MSE in Figure 1 of the global rebuttal PDF.

Statistical significance tests.

Thank you for pointing out the necessity of a significance test. Across 48 settings of the benchmark, our method outperforms 10 baseline methods in most cases (Table 6). Considering the diversity of the datasets (covering electricity, traffic, energy, and climate) and the diversity of the comparison methods (including linear, MLP, CNN, and Transformers), this sufficiently demonstrates the superiority of our proposed method. We also demonstrate the stability under different random seeds in Table 9, indicating that the performance is not coincidental. To alleviate any concerns, we have supplemented our analysis with t-test significance experiments. Due to the large number of experimental settings and the lengthy training time of some methods, we selected iTransformer and PatchTST, which have performances most comparable to ours, for comparison. We repeat 10 times for each method and each setting. The results are as follows. The T-Statistics < 0 and p-value < 0.05 are marked in bold.

Our method significantly outperforms the previous SOTA methods in most cases.

The results of Lookback Window Length 720.

We extend the lookback window analysis from range [48, 336] to [48, 720] and show the results in Figure 1 of the global rebuttal PDF. The figure shows that SOFTS can also achieve superior performance when the lookback window length is extended to 512 and 720.

Training time for each method.

Thank you very much for your suggestion. However, training time is influenced not only by the complexity of the model itself but also by the training strategy. Earlier methods, such as FEDformer and Informer, often completed training early due to early stopping strategies to prevent overfitting. These methods have shorter training times than current SOTA methods but perform much worse. To ensure a fair comparison, we comprehensively present model performance, inference time, and memory in Figure 3, where the latter two are only related to the complexity of the model. Therefore, we believe this figure more fairly demonstrates the superiority of our method in terms of performance and efficiency compared to merely showing training time.

It is unclear the MLP and Linear operations are channel independent or dependent.

Yes. To make it clear in the paper, we have specified every MLP in the form of mapping. For example, $\operatorname{MLP}_1: R^{d} \mapsto R^{d'}$ in line 147 means it project the axis with dimension $d$ to dimension $d'$. This mapping is shared across other axes including the channel dimension. Therefore, it corresponds to channel independence. They function similarly to the FFN layer of the Transformer.

The sentence "... rely too heavily on the correlation to achieve satisfactory results under distribution drift" in the Abstract is not clearly explained.

This sentence corresponds to lines 33-35 in the introduction "However, such channel mixing structures were found vulnerable to the distribution drift". For rigor, we change it to "fail to achieve satisfactory results under distribution drift when incorporating the correlation".

Minor mistakes or typos: Embedding module is missing in Figure1; oi ∈ Rd' should be oi ∈ RC×d' in Algorithm 1.

Thanks. We omitted the embedding module in the figure to spare space, and we will add it to the figure in the final version; $o_{i}$ is the global core representation for the whole multivariate series. It is not channel-specific, so it should be $o_{i} ∈ R^{d'}$ and it is not a typo.