TwinsFormer: Revisiting Inherent Dependencies via Two Interactive Components for Time Series Forecasting

Thanks for your time and effort in reviewing our paper. We are pleased to see that you identified the experimental results. However, we guess that there are some misunderstandings about the novelty of our work, which we try to clarify in our following responses to specific questions and concerns:

Generally, time series forecasting aims to predict future temporal variations based on historical values of time series, where the primary challenge is how to effectively capture inherent dependencies from historical data. Inherent dependencies refer to the correlations and patterns within the data that are crucial for accurate prediction, such as temporal dependencies, seasonal and trend patterns, autocorrelation, multivariate correlation, and so on. More specifically, temporal dependencies are the influence of past values on future values. For example, today's weather can be predicted based on yesterday's weather conditions. Seasonal patterns denote repetitive patterns that occur at regular intervals, such as daily, weekly, or yearly cycles. Trend patterns are long-term upward or downward movements in the data. Autocorrelation is the correlation between a time series and a lagged version of itself. Multivariate correlation is the correlation among multiple time series.

In time series forecasting, trend-seasonal decomposition is a common technique to tackle intricate temporal patterns. In lines 053-084 of our manuscript, we point out that existing decomposition designs generally utilize two independent branches to highlight seasonal and trend properties separately for final prediction. Considering trend-seasonal decomposition is an untrainable linear transformation (i.e., moving average kernel), the decomposed trend and seasonal components cannot accurately characterize the inherent dependencies of the raw time series.

To intuitively understand the limitation of the seasonal-trend decomposition, we provide the trend-seasonal decomposition visualization of two channels (i.e., variates) on the Electricity and Traffic datasets. As seen in Figure 2 of our manuscript, the "Observed" denotes the raw time series, and the "Trend" and "Seasonal" are decomposed components from the raw time series by the moving average kernel. Comparatively, the trend and seasonal components exhibit distinct characteristics for the observed time series, where accurately capturing the inherent dependencies from the seasonal or trend components alone is impossible. Taking channel 11 on the Electricity dataset as an example, the long-term upward or downward movements roughly depicted by the trend component of channel 11 cannot reflect the periodic patterns of raw channel 11, while the repetitive patterns depicted by the seasonal component of channel 11 are not actual temporal patterns of channel 11. Based on the observation, we believe the interactions between decomposed components are important for time series forecasting.

Benefiting from the ability to model long-term dependencies (i.e., autocorrelation and multivariate correlation) by the attention module, Transformer-based methods have shown significant success in time series forecasting. To learn the interactions between decomposed components, we take the attention mechanism as the basic module to connect the seasonal and trend components. To avoid excessive computational overhead caused by the attention mechanism, we must consider how to obtain the multivariate correlations for different components. In Figure 2 of the manuscript, we observe that the seasonal components can better reflect the temporal patterns of observed time series, so we use the seasonal components to learn the multivariate correlation of time series data. Based on Equation (1) of the manuscript, we can find that the seasonal component is obtained by subtracting the trend component from the original time series, so we replace the addition operation of the existing Transformer-based architecture with a subtraction mechanism. In lines 243-269 of our manuscript, we provide a rationality analysis for our dual-stream framework.

In summary, by delving into the trend-seasonal decomposition process and combining the attention mechanism's remarkable ability to learn long-term dependencies, our proposed Transformer-based dual-stream interaction framework is traceable rather than heuristic. Based on our interactive architecture, we can learn inherent dependencies beyond the multivariate correlation to improve time series forecasting. Regarding the design choices, we acknowledge that some may initially appear ad-hoc. However, we would like to clarify that these choices were made based on a combination of empirical evidence and theoretical considerations. While some decisions were indeed informed by preliminary experiments, they were also guided by established principles in the field.

Dear Reviewer pkJh,

Sorry to bother you again.

Given the importance of timely communication and the rebuttal phase nearing the end, we would like to know if we have addressed your concerns. If you have any remaining questions, please let us know. We are looking forward to your valuable reply.

Thank you for your efforts in our paper.

Best regards,

Authors

These results exhibit that the performance of TwinsFormer is stable in ablation studies.

Thank you for your time in reviewing our paper and the detailed comments. The followings are our responses to your concerns and questions:

W1: Difference to DESTformer.

DESTformer is an interesting work that fuses the encoded features of decomposed components at each layer of the model via the addition operation. However, our method and DESTformer are completely different. 1) Since we encode the historical time series of each variate into an embedding, our seasonal-trend decomposition acts in the observed time series space rather than the embedding space of DESTformer. 2) DESTformer is designed to alleviate the information utilization bottleneck of existing decomposition-based transformers, while TwinsFormer is the first attempt to explore the interactions between decomposed components to break through the limitation of the seasonal-trend decomposition.

W2: Different performance to iTransformer.

In model variate ② of Table 3, we feed trend components to the seasonal branch and feed seasonal components to the trend branch. The similar performance to TwinsFormer only indicates that swapping inputs for different branches has relatively little impact on model performance. To further illustrate the role of our interactive module in our framework, we disable our interactive module in the trend branch as the model variate ⑧. Specifically, we feed the seasonal components obtained by the moving average kernel to the seasonal branch and treat the trend components obtained by the moving average kernel as the output of the trend branch. The results of ⑧ on ECL, Traffic, PEMS03, and PEMS07 datasets are listed below:

Without the interactive module, the performance degradations of model variate ⑧ are obvious, even worse than iTransformer. These degradations indicate that despite iTransformer providing a good backbone for our approach, our interactive module plays a key role in further improving forecasting performance.

Q1: Considering that an untrainable linear transformation (i.e., the moving average kernel) cannot reflect the non-linear pattern of the observed time series, we propose a Transformer-based interaction framework to better capture the intrinsic dependencies for time series forecasting. Since we utilize the model to learn the interactions between decomposed components for time series forecasting, it is significant to analyze the impact of the inputs and operations of key modules on the model performance. The results in Table 3 are obtained with the same fixed random seed, which means the result is the same in each run under the same hyperparameters. Since the variance is too small, we compute the standard deviation of ablation studies with five random seeds as below:

Dear Reviewer ioRG,

Sorry to bother you again.

Given the importance of timely communication and the rebuttal phase nearing the end, we would like to know if we have addressed your concerns. If you have any remaining questions, please let us know. We are looking forward to your valuable reply.

Thank you for your efforts in our paper.

Best regards,

Authors

Anyway, thank you for your prompt reply.

In this paper, we delve into the trend-seasonal decomposition and find that the seasonal and trend components obtained by the trend-seasonal decomposition cannot accurately capture the inherent dependencies of the time series due to the untrainable linear transformation (i.e., the moving average kernel).

Therefore, we propose a first Transformer-based framework that explicitly explores inherent dependencies by learning implicit and progressive interactions between different components for time series forecasting.

Concretely, we design an interactive module around the attention module and the feed-forward network to learn more robust and reliable decomposed representations based on residual and interactive learning.

Technically, we only utilize the trend-seasonal decomposition to initialize our seasonal and trend components, and our dual-stream interaction framework can enhance the transformation and decoupling of seasonal and trend.

To better understand the learning process of the seasonal and trend components, we provide more visualization results in Figures 10-12 of the revised manuscript.

Experimentally, TwinsFormer achieves state-of-the-art performances in both long-term and short-term forecasting tasks, and is plug-and-play into existing Transformer-based methods with negligible extra computational overhead.

We understand your concern regarding the novelty of our method. To better address this issue, we would greatly appreciate it if you could provide more detailed feedback on the specific aspects of our approach that you find lacking in novelty. Your insights would help us understand how we can further enhance the originality and impact of our work.

We look forward to your further feedback and are committed to addressing any concerns raised by you.

Many thanks to Reviewer AH3t for providing a detailed review and insightful questions. We address your concerns here:

W1: The metrics used in time series forecasting, Mean Squared Error (MSE) and Mean Absolute Error (MAE), are highly sensitive to small changes, especially when the baseline models are already performing at a high level. A difference of less than 0.02 in these metrics can make a significant improvement in practical applications, particularly in domains where even minor errors can have substantial consequences, such as financial and traffic forecasting. Furthermore, All experimental results were obtained using the same random seed, meaning that running the experiments with the same parameters consistently yields identical results. Additionally, the results in Tables 1 and 2 are the average results of four different forecasting lengths, and we provide the full experimental results in Tables 6-7 of the manuscript.

W2&Q1: To ensure the fairness of the experiment, all experimental results are obtained by running under the same random seed. Meanwhile, we reported the standard deviations of TwinsFormer performance in Table 8 of the manuscript. We also provide statistical tests on ECL and all subsets of ETT and PEMS with five runs of random seeds. The performance is averaged from four prediction lengths. The standard deviations and T-test statistics (MSE|MAE) of TwinsFormer and iTransformer are listed below, showing that the performance is on par with the previous SOTA iTransformer within the margin of error.

W3: In the rationality analysis of Section 3, we formulate the model to provide a formal expression to facilitate the understanding of the practical significance of our dual-stream structure and its compatibility with the decomposition design. Replacing concatenation with addition operations is intended to simplify the formula of the interactions between decomposed components rather than highlight which operation is the optimal design choice. To better illustrate the difference between the addition and concatenation operations, we compare the average performance (MSE|MAE) of the two operations on three datasets as below.

The results indicate that the performance of '+' is inferior to that of '[]', which may be due to the addition operation inhibiting the decoupling of seasonal and trend components.

W4&Q2: The performance (MSE|MAE) of univariate time series forecasting on all the subsets of ETT is presented below. As iTransformer, TimeMixer, and Crossformer do not offer training hyperparameters and results for univariate forecasting tasks, we compare our method with PatchTST, DLinear, and FEDformer.

Thanks for reviewing our paper. The following are our responses to your specific questions and concerns:

W1: In the manuscript, we highlight the interactions between decomposed components (i.e., seasonal and trend components) rather than the new design for the decomposition. Specifically, we employ the moving average kernel to decompose the time series data into seasonal and trend components, and then design a Transformer-based dual-stream framework to make different components learn from each other and capture better inherent dependencies for the final forecasting. In addition, we supplement the experimental results of wavelet and non-linear decompositions as described by Reviewer WMfY, which illustrates the compatibility of our dual-stream framework for more decomposition designs. As an MLP-based framework, TimeMixer introduced the decomposable multiscale mixing module to learn the interactions among different scale seasonal (or trend) components rather than the interactions between seasonal and trend components. Existing decomposition designs treat the seasonal and trend components as two unrelated variables, ignoring the interactions between the decomposed components. Therefore, TwinsFormer is the first attempt to consider interactions between decomposed components on Transformer for time series forecasting.

W2:: Indeed, different lookback window lengths affect the robustness of the decomposed trend and seasonal components. We analyzed the lookback length sensitivity in lines 408-427 of our manuscript, and Figure 4 clearly illustrates that our TwinsFormer can effectively capture inherent dependencies from a longer lookback window.

W3: Generally, the most important module of Transformer-based forecasters is the attention mechanism. In Table 4 of our manuscript, we mainly analyze the impact of different attention mechanisms on the performance of our framework. For PatchTST and Crossformer, the time series fed to their attention modules is obtained by patching and segmentation operations, which cannot capture the seasonal and trend characteristics of the time series. As for iTranformer, its success lies in learning inherent dependencies from the dimension of variates rather than exploring new attention mechanisms. Without the decomposition and interactive modules, our backbone is consistent with iTransformer. Therefore, we did not analyze the attention compatibility and performance promotion on PatchTST, iTransformer, and Crossformer in Table 4.

W4: In our work, we adopt channel independence to decompose the lookback length of each channel into seasonal and trend components, and then encode the decomposed components into corresponding embeddings. We utilize the attention mechanism in the seasonal branch to learn the multivariate correlation among different seasonal embeddings. Meanwhile, we treat the residual seasonal signals obtained by the subtraction mechanism as the supervision information to perform interactive learning with the trend embeddings in the trend branch. In other words, our interactions between decomposed components among multiple channels.

W5: Both iTransformer and TimeMixer are excellent works, and we adopted some of their experimental formats because their approaches to model analysis are highly representative and widely recognized in the field. Moreover, our dual-stream Transformer-based framework incorporates both decomposition design and a plug-and-play interactive module, necessitating an exploration of the individual components' roles and the generalization capabilities of our module. While the analysis format may resemble existing methods, our primary focus is on the unique contributions and effectiveness of our dual-stream Transformer-based framework.

Q1: We regard the decomposed components obtained by the moving average kernel as the initial decomposed components, and then feed the encoded seasonal and trend embeddings to the TwinsBlock. To verify our TwinsBlock can still adequately deal with the trend and seasonal components, we supplement additional visualization results in Figure 10 of the revised paper. In Table 3 of the manuscript, we conducted the ablation study on swapping seasonal and trend components (i.e. the model variant ②), and the results show that feeding the seasonal components to the attention module is more beneficial for model performance. Recall Figure 2 of the manuscript, the trend components are very smooth over a certain time interval, while the seasonal components can better reflect the temporal patterns of the observed time series, which also suggests that the seasonal components can better represent the multivariate correlations. Thus, we do not need to consider randomly swapping the trend-seasonal inputs to a TwinsBlock in the middle of the model.

Dear Reviewer eGh2,

Sorry to bother you again.

Given the importance of timely communication and the rebuttal phase nearing the end, we would like to know if we have addressed your concerns. If you have any remaining questions, please let us know. We are looking forward to your valuable reply.

Thank you for your efforts in our paper.

Sincerely,

Authors

Many thanks to Reviewer WMfY for providing thorough and insightful comments. The following are our responses to your specific questions and concerns.

W1: Thanks for your scientific rigor. We compare the performance (MSE|MAE) of the moving average kernel (MAK) with Wavelet (Wav) and Non-linear (Non) decomposition designs on three benchmarks below:

These results show that our method has good compatibility with various decomposition techniques.

W2: Thanks for your valuable comments. Since the existing decomposition designs uniformly use the kernel size of 25 by default, we ignore the decomposition mechanism's hyperparameters. We provide the results (MSE|MAE) related to the kernel size (KS) and embedding dimension (Dim) on ECL and Traffic below:

W3: To comprehensively analyze the computational cost, we treat TwinsFormer with 20%, 60%, and 100% variates as small, medium, and large model sizes, respectively.

We can observe that our TwinsFormer requires less memory and runs faster than other Transformer-based models.

W4: According to Equation (1) of the manuscript, we can find that the seasonal components are obtained by subtracting the trend components from the original time series in the trend-seasonal decomposition. To better learn the interactions between seasonal and trend components, we believe that the subtraction mechanism is more in line with the principle of trend-seasonal decomposition and further facilitates the decoupling of seasonal and trend components. Since the concatenation will change the channel dimension of the encoded features and bring additional computational overhead, existing Transformer-based models use the addition to implement the skip connection by default. Therefore, Transformer-based methods do not use the concatenation to implement the skip connection. In Table 3 of the manuscript, the model variant ③ shows the performance of replacing the subtraction mechanism with the addition.