Rethinking the Temporal Modeling for Time Series Forecasting with Hybrid Modeling

We would like to express our gratitude for your patient review of our paper and the valuable problems you provided. Indeed, our model design is relatively simple, but compared to other methods that involve the coupling of multiple modules, this simplicity allows for a more straightforward evaluation of the effectiveness of each module, thereby providing a more intuitive representation of the efficacy of our model. Here is a detailed response addressing your points:

1. As we stated, this study started from the strong motivation for the decoupling of modules in previous works and a more refined analysis in previous works, as in previous studies on time series forecasting models, many modules were indeed coupled. For instance, works like Autoformer and Fedformer utilized techniques autocorrelation mechanisms, position embedding, and temporal embedding, e.t.c.; In PatchTST, mechanisms such as ReVIN, channel independence, patching, and attention mechanisms were introduced. These models are products of various coupled module methods, making it challenging to directly understand the source of performance in these models. Therefore, in our experiments, we found that comparing the performance between models with coupled modules was unfair and could lead to misleading conclusions and inaccurate model designs. We addressed this by decoupling the main modules of the models and quantitatively analyzing their impact through ablation experiments, a point not previously discussed in the literature.

2. Building on the motivation mentioned above, we identified some possibly unnecessary designs, such as the excessive pursuit of attention modeling for time series forecasting models. In comprehensive ablation experiments, we confirmed that the design like introducing context embedding and attention mechanisms might unnecessary. Furthermore, we delved deep, quantitatively and comprehensively evaluated the designs of some recent MLP models, indicating their strong performance in temporal prediction tasks. Based on the observations mentioned above, we decoupled various modules and parameters within a hybrid modeling framework, illustrating how combining these elements and hyperparameters is a crucial theme. We appreciate your identification of shortcomings in our paper, and in the revised version, we provided a more concise and direct description to enhance readers' understanding of our contributions.

We would like to further highlight the core contributions of our paper in the revised manuscripts. And once again, thank you for your review, and we look forward to any further suggestions you may have, and we would like to express our gratitude for your thorough review of our paper and the valuable insights you provided.

Q2: Is there a reason why patchTST working so much better than transformer achitetures like AutoFormer/Fedformer, especially, where the ablation studies clearly show attention, PE mechanisms are useful?

Ans: You've raised a very good point, and indeed, as mentioned earlier, the decoupling of modules in previous works and a more refined analysis are the main motivations proposed in this paper. Both AutoFormer and Fedformer are highly innovative works, and we believe the core issue causing this confusion is: why are AutoFormer and Fedformer so effective? On this matter, we understand it is also a key argument in this paper: were these previous works truly compared fairly? For instance, when comparing AutoFormer and Fedformer with the baseline model Informer, there is a significant coupling of modules in the comparison between models. This fails to reflect which modules truly influence the model and contribute to performance improvements/degradations. In our experiments, we observed that the inclusion of positional encoding actually leads to a decline in model performance. Furthermore, we may mention that the recent and representative work DLinear, that did not use positional encoding modules, achieved significantly better performance than AutoFormer and Fedformer on time series forecasting tasks using only a simple linear model. Therefore, we posit that the perceived effectiveness of position encoding in AutoFormer is, to a large extent, due to insufficient testing of its parameters and modules (we believe the effectiveness of positional encoding is not discussed by the authors in the original paper). The coupled modules contribute to the perception of their effectiveness, which, in reality, may not be the case.

Thank you for your valuable feedback and constructive comments on our paper. We really appreciate the time and effort you dedicated to reviewing our work. Below is a detailed response to your points:

W1: Is there any particular reason to not compare this model with the state-of-the-art long-range forecasting model like Spacetimeformer (Grigsby et al. 2023), and NBeats. Both models perform vey well for long range forecasting on multivariate data. SpaceTimeFormer paper shows result for prediction length upto 672 (on some weather data). & Q2: Can you show the results compared to SpaceTimeFormer and Nbeats? SpaceTimeformer model also has local+global architeture approach. & W2: Can you show the results compared to SpaceTimeFormer and Nbeats? SpaceTimeformer model also has local+global architeture approach.

Ans: Thank you for your suggestion! We have supplemented the paper with additional baselines, including SpaceTimeFormer, N-BEATS, and N-BEATS' following-up work N-HiTS, and two new baselines which combining RevIN and linear models, as mentioned by another reviewer in Appendix A.3.

Regarding your mention that the SpaceTimeFormer model also has a local+global architecture approach, it is worth noting that the emphasized local/global structure in the SpaceTimeFormer model is achieved by computing attention between multiple variables at each time step (local attention), while global attention is calculated directly across the entire temporal sequence. We may interpret the local attention in this approach as feature fusion performed per time step using the attention module, while global attention is similar to the attention calculation in the temporal dimension used in PatchTST. In our experimental results, we found that SpaceTimeFormer did not achieve comparable performance to UniTS and PatchTST. We attribute this to our consistent observation in experiments that the introduction of attention and context embedding did not improve model performance. Further analysis of the results in SpaceTimeFormer is needed, as they also emphasized in their paper that the introduction of the RevIN module effectively improved model performance. However, when compared with baseline models such as AutoFormer, they did not introduce the same normalization modules to these models. Taking a broader perspective, we observed that in previous works, due to the coupling of too many modules, analyzing only individual modules in experiments often cannot well explain the results, which is also a core point raised in this paper.

As for the other baseline N-BEATS you metioned, we further provided the results of N-BEATS and its following-up work N-HiTS in the revised manuscript. They both adopt a design combining residual stuck with linear networks. However, these two models do not use the RevIN module, and in the final results, their predictive performance is lower than the two purely linear models we introduced, namely RMLP (RevIN+MLP) and RLinear (RevIN+Linear). Thus, this decoupling of modules in previous works and a more refined analysis are the main motivations proposed in this paper. We aim to unravel the intricacies of prior research, seeking the optimal and most suitable modules for constructing time-series prediction models. By untangling the intricacies of previous works, we hope to assist the time-series forecasting community in gaining a more fundamental and in-depth understanding of the various existing models.

Q4&Q5:The hyperparameter selection (including learning rate, hidden size, e.t.c. besides the use of lookback window length) is only used for the proposed UniTS but not for other baselines. I am doubting whether it results in another unfair comparison problem; There is no complexity analysis, especially, the complexity introduced by the hyperparameter selection.

Ans:Your observation on the necessity of discussing complexity is well-founded. In our paper, In our discussions, we proposed a method that essentially decomposes modules previously coupled in various temporal forecasting models. We discuss which of these modules are useful and how the choice of modules affects the model. This makes parameter selection, previously overlooked in the literature, an unavoidable issue. In previous discussions, the performance of models presented is often the result of manual selection on the given architecture, making it a core issue in our paper.

Analyzing this issue directly is challenging for two reasons: first, we cannot exhaustively explore all possibilities for each model, and second, conducting a parameter search in a given parameter space would introduce human bias, leading to potential unfairness. However, based on their performance on real-world datasets, we can propose a comparison method in limited scenarios. In practice, the most effective time series prediction models do not employ overly complex network structures or high-dimensional parameters. Through our experiments, we observed that overly complex parameterized structures often have a negative impact on performance. From the perspective of the dataset used and the models employed in this paper, if our goal is to find an optimal model, we can define a search space. In actual experiments, we found that for almost all models, we only need to search within a relatively small hidden size and a shallow network range. Therefore, for all models, the number of times they select optimal network parameters is relatively small. From this perspective, in cases where the search space difference is not significant, in terms of computational complexity, the cost of parameter search mainly comes from the training cost of each parameter configuration, which can be further quantified by the training cost per epoch for each model. From this point, we provide a further discussion of model complexity in Appendix A.4 in the revised manuscript.

It is important to note that the comparison we provide is not perfect and does not represent absolute fairness. However, we have demonstrated that this search is simple and feasible within the existing knowledge. In a limited time cost, using the structure we propose, we can train models that achieve effective time series prediction performance.

Dear Reviewer,

Thank you for your thorough review and valuable feedback on our paper. We appreciate your insights and have carefully considered your suggestions. Here is our response to address the key issues you raised:

Q1: Some parts of the proposal UniTS are not introduced clearly

Ans:We will provide additional clarity on the meanings of H^{l,N} and H^{g,N} at the caption of Figure 1, ensuring a more comprehensive explanation in the revised manuscript. Regarding how the data is passed to the GFE and LFE modules after patching, we have provided a connecting description in the main text. You can find it highlighted in blue in section 3 of the revised manuscript. Additionally, we have further corrected the typos and wording issues in the original text that you mentioned, which could potentially be misleading.

Q2: The RLinear and RMLP models in the following reference outperform PatchTST on some datasets. It is better to compare with them as well.

Ans: In light of your and another reviewer's suggestions, we have incorporated the results of RLinear, RMLP, and two additional baselines in the revised version of the paper (Appendix A.3). Both RLinear and RMLP underscore the effectiveness of combining ReVIN with linear networks, showcasing superior performance to PatchTST on widely used datasets. These findings align seamlessly with our experimental observations. The additional experimental results have been appended, and we present them in tabular format for your convenience.

Q3:I think it is unfair to compare with PatchTST/64 which uses lookback window length 512 only. As shown in the Figure 2 of the PatchTST paper, the performance is also changed with different lookback windows. It is better to choose the best results from different lookback windows for PatchTST as well, since this paper highlights the unfair comparison issue.

Ans: Thank you very much for pointing out the issues covered in the description of our paper. Indeed, in our experiments, we performed parameter search for each baseline model as well. For DLinear and PatchTST, we essentially reproduced the results reported in their respective papers. However, as you rightly pointed out, directly using values such as the lookback window size of 512 from the original PatchTST paper could lead to suboptimal results and raise concerns about fairness. Thus, it is essential to clarify this aspect, as it directly relates to the fairness of performance comparisons between various models.In the revised manuscript, we use the optimal results obtained through parameter search in our own replication as the results in Table 1. We have also clarified how the data in Table 1 was obtained in section 4.1 to ensure a more fair perspective for the result studies of our experiments.