A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle

Thank you very much for your comments. We truly appreciate your effort and time for reviewing our work.

Response to the comment challenging the assumption that the model's output is deterministic:

     While MSE is linked to the assumption of Gaussian noise during training, it does not make the model's predictions stochastic. In this case, the model trained with MSE produces a single, fixed output for a given input. To obtain truly probabilistic or stochastic outputs, one would need to model and sample from the predictive distribution explicitly, such as through Bayesian methods or by estimating parameters of a distribution and sampling accordingly.

Response to the comment regarding pre-trained Transformers such as Time-MoE and Chronos:

     As stated at the end of Sec. Related Work, our study focuses on lightweight transformer architectures that are designed to be trained from scratch for an individual dataset. We did not include pre-trained models as they typically follow different training protocols and involve more complex architectures, making fair and transparent comparisons difficult. Additionally, Time-MoE uses a decoder-only architecture, which is fundamentally different from the encoder-based structures used in the lightweight models. Chronos, on the other hand, tokenizes time series into discrete bins through simple scaling and quantization, a strategy that differs significantly from the point-wise, patch-wise, and variate-wise tokenization. We appreciate the suggestion and will include references in the revision.

Response to the comment regarding broader validation of the importance of skip-connection:

     We conducted the skip-connection ablation study to explore the following questions (Line 327): “Why do transformers with basic attention mechanisms perform well in time series forecasting? Which components of the basic transformer architecture contribute most to this success?” Our focus is specifically on basic and effective transformer architectures. iTransformer meets both criteria, and we consider its structure representative for understanding key design components in transformer-based models. We analyzed skip connections in iTransformer because we hypothesized that they play a key role in learning intra-variate patterns, especially within an architecture designed to support inter-variate attention. Moreover, in models using point-wise tokens, skip connections primarily enhance inter-variate patterns, which differs fundamentally from the goal of this ablation study.

Response to comments regarding additional references.

     Thank you for your suggestions. While we agree that the suggested papers are relevant to the broader topic, we believe that the specific findings of our work are not addressed in them.

     [1] Rethinking Channel Dependence... ICLR 2024

     [1] proposes LIFT, a plugin method designed to identify and utilize locally stationary lead-lag relationships between variates to improve forecasting performance. While it focuses on modeling lead-lag dependencies, our work evaluated the effectiveness of various models in capturing general inter-variate relationships. The mutual information-based scores we propose are model-agnostic and capable of assessing inter-variate dependencies without being limited to specific relationship types.

     [2] Reversible Instance Normalization... ICLR 2022.

     Reversible Instance Normalization (RevIN), a method similar to Z-score normalization, was designed to address distributional shifts in time series data. A key difference is that RevIN incorporates learnable parameters within its normalization process, whereas the Z-score normalization examined in our study is the standard, non-learnable version. Moreover, our findings contradict the suggestion that "Z-Norm addresses distributional shift." Specifically, we observed that Z-score normalization degraded model performance on synthetic datasets which are non-stationary with monotonic trends (Table 4).

Response to other suggestions and questions:

     1. "...include error bars", "...without multiple runs", "...provide training configurations".

     As stated in Sec. Experiments (Line 216 - 219): "All the experimental results in this work are averaged over 3 runs with different random seeds. Standard deviation of the results are provided in the Appendix." Both Fig. 3 and 4 include error bars. We will add hyperprameter configurations in the appendix in the revised manuscript.

     2. "...provide showcases of synthetic datasets".

     We will add visualization of synthetic datasets in the revised manuscript.

Thank you very much for your thoughtful comments. We truly appreciate your effort and time for reviewing our work.

Response to comments regarding additional metrics for evaluating forecasting accuracy:

     In this work, our primary focus is on understanding how different token representations within the attention mechanism lead to significant variations in model performance. We report MAE and MSE because they are the most widely used evaluation criterion in the literature, allowing for easy comparison with the related works. We acknowledge that this is not sufficient to evaluate models comprehensively. This is also the reason we introduce several mutual information-based scores in this work to explore model behaviour in capturing interactions between variates. The broader question of how to design and apply appropriate evaluation metrics for time series forecasting is a substantial topic in itself, requiring holistic analysis of both datasets and model behaviour. We consider this a valuable direction for future research.

Response to comments regarding the inclusion of additional datasets:

     We fully agree that the benchmark datasets commonly used for evaluating time series forecasting models should be significantly expanded. This is one of the key points we aim to highlight through our work. In this study, we selected the most widely used datasets in the literature, as they have become a standard reference for comparing model performance. However, based on our analysis, these benchmarks are limited in scope and do not sufficiently reflect the diversity of real-world forecasting challenges. To address this, we present results not only on these standard datasets but also on eight synthetic datasets and two real-world healthcare datasets, which help reveal the limitations of relying solely on several homogeneous benchmarks. We appreciate the suggestion and will examine the recommended datasets in our experimental settings.

Response to comments regarding essential references:

     Thank you for the valuable suggestion. We agree that considering dataset characteristics and appropriate modelling strategies—like traditional methods such as ARIMA—can greatly benefit modern time series forecasting. We see this as an important direction for future work, which should ideally be addressed alongside the analysis of dataset characteristics and the design of suitable evaluation metrics. We'll acknowledge this point in the revised manuscript.

Response to questions:

1. Limited Complexity -- Why only two variates included in the synthetic data?

   We used only two variates in the synthetic data to maintain clarity in analyzing inter-variate dependencies. We had considered to generate more variates, however, it made the analysis of results much less clear. For instance, introducing a third variate would require defining its dependency on the other two, resulting in two additional hyperparameters. Moreover, since the second variate is already dependent on the first, the dependency of the third variate would effectively be influenced by three parameters. This added complexity would make the analysis less transparent and interpretable.

2. Simplified Dependency Structure -- Why only linear dependencies between variates in the synthetic data?

   Similarly, linear dependencies are straightforward to interpret and analyze. We had considered increasing the complexity of the dependencies if the models demonstrated strong performance in capturing linear relationships. However, even with these simple linear dependencies, we observed that patch-wise and variate-wise models still rely heavily on intra-variate dependency except in cases where the inter-variate dependency is extremely high (e.g. $\alpha = 0.8$). Given this, we did not see a clear benefit in introducing more complex dependencies at this stage.

Thank you very much for your encouraging comments. We truly appreciate your effort and time for reviewing our work.

Response to questions:

1. Why use transformers?

   In previous work on developing new transformer architectures for time series forecasting, there has been little validation of whether the attention mechanism functions as intended. Although several studies aim to enhance transformers' ability to capture inter-variate dependencies, none have explicitly verified this in relation to the underlying data characteristics and model outcomes. This lack of validation motivated us to explore the issue in our study.

   Our findings show that patch-wise and variate-wise transformers effectively capture intra-variate patterns, which remains a useful property in time series forecasting. Additionally, experiments with synthetic datasets demonstrate that certain transformer architectures—such as Crossformer—are capable of learning inter-variate patterns. This suggests that transformers can still be effective in scenarios where such patterns are important. We hope that our findings can offer useful insights for designing new architectures or for more effectively applying transformers in related applications.

2. Is attention useful in iTransformer?

   As we understand it, attention is still useful in iTransformer. For example, in our skip-connection ablation study, we observed that iTransformer performed better without skip connections on synthetic datasets with high inter-variate dependencies. This suggests that, in such scenarios, the attention mechanism alone is still capable of capturing meaningful patterns. Additionally, the skip connection in iTransformer is added to the attention output—not as a simple bypass, but rather as a means to reinforce the self-attention for each variate.

Thank you very much for your encouraging comments. We truly appreciate your effort and time for reviewing our work.

Response to Claims And Evidence:

1. In Table 3, removing the skip connection leads to a clear performance drop across all benchmark datasets, with particularly significant degradation on Electricity (MAE increases from 0.266 to 0.320) and Traffic (MAE increases from 0.283 to 0.591). On the synthetic datasets, this degradation becomes negligible while $\alpha$ increases under the condition of $\gamma$ = 0.95. In the manuscript, we stated that "removing the skip connection notably degrades performance across most datasets, except for synthetic datasets with high inter-variate dependencies ($\alpha$ = 0.8) or low autocorrelation ($\gamma$ = 0.5)," which aligns with your observation that "low autocorrelation does not seem to be degraded at all." We will revise this statement to improve its clarity in the revision.

2. Apologies for the confusion—this statement was incorrect. It should be: "It improves performance on synthetic datasets with high inter-variate dependencies ($\alpha$ = 0.8) under low autocorrelation ($\gamma$ = 0.5)." We will revise the corresponding claim to: "The variate-dependent decoder has the potential to enhance the model’s ability to capture inter-variate interactions, particularly in scenarios with strong dependencies between variates."

Response to broader literature and essential references:

     Thank you for your valuable suggestions. We will add discussions of both papers in the Related Work section.

Response to questions:

1. Why was TFT not included?

   In this study, we focused on representative transformer architectures from the perspective of token representations within the attention mechanism. TFT, however, is a hybrid model that incorporates variable selection and LSTM-based encoders prior to applying attention. As a result, its token representations are less explicit and not as directly comparable to the models we selected. We agree that TFT is a relevant model and will include a discussion of it in the revised manuscript.

2. What is the major role of MAX MI?

   We propose MAX MI as a measure of the mutual information captured by a model between the most strongly interacting variates. This metric helps assess whether a model is effectively learning inter-variate dependencies, particularly in cases where the number of variates is large and only a few exhibit strong interactions. For instance, in Figure 3, Crossformer achieves the highest MAX MI on the Traffic dataset (which has 862 variates), while its average inter-variate mutual information (Avg. Inter MI) remains lower than that of most other models.

3. Explanation of $\sigma_{ij}$?

   $\sigma_{ij}$ estimates the extent to which changes in the prediction of variate $j$ are caused by changes in variate $i$. Unlike Pearson's correlation, it is more versatile as it captures both linear and non-linear relationships.
   Mathematically, $\sigma_{ij}^2$ is the variance of the predictions of variate $j$ conditioned on inputs $\mathbf{x}$ where all variates are held fixed except for variate $i$. To estimate this conditional variance, the variation of variate $i$ is introduced through N different samples by augmenting original samples in the dataset. Specifically, each original sample is augmented into N = 5 versions, differing only in the value of variate $i$: one instance is set to zero, one retains the original value, and the remaining instances are generated by adding Gaussian noise of varying strengths to the original value. We will amend the explanation in the revised manuscript to improve the clarity of the paper.