Timer-XL: Long-Context Transformers for Unified Time Series Forecasting

Thank you again for your comments, which are very helpful for us to improve the quality of the paper. We explored the properties of TimeAttention and enlarged zero-shot evaluations. If you have any further questions, we are looking forward to discussing with you.

Many thanks to Reviewer V4mu for providing a detailed review and recognizing our contributions.

Q1: How does the order of the flattened sequence influence the equations?

It is a very insightful question. Given $N$ time series, TimeAttention has the following properties:

• Avoid Permutation-Invariance on the Temporal Dimension: Self-attention is permutation-invariant, which contradicts the causality in the temporal dimension.
• Permutation-Equivariance of Multiple Variables: Formally, $\operatorname{Attn}(\mathcal{P}(V1,...,VN)) = \mathcal{P}(\operatorname{Attn}(V1, ..., VN))$, where $\mathcal{P}$ is any permutation of $N$ variables. With this property, the input and output of variables have a correspondence in position: Shuffling the input order of variables should be reflected in the output order of variables but not affect the temporal causality.

$\underline{\text{Equation (6)}}$ conforms to the above property: First, we use RoPE (indexed by $i, j$) on the temporal dimension to avoid permutation-invariance (we empirically highlight the advantage of RoPE over other embeddings in Section 4.5). Second, $u$ and $v$ (judged by $m, n$) are used to distinguish diagonal elements (endogenous variables) from non-diagonal elements (exogenous variables). The permutation-equivariance of variables can be satisfied (this work [1] provides a detailed proof).

Therefore, $\underline{\text{Equation (6)}}$ is unaffected if the variable order of the flattened sequence is changed by $\underline{\text{Equation (4) and (5)}}$. The variant of variable order actually affects the formulation of the mask in $\underline{\text{Equation (8)}}$. In our paper, we use the temporal-first approach (i.e., two series are divided into six patches indexed by A: 1 2 3, B: 4 5 6, as shown in $\underline{\text{Figure 2}}$), the mask is formulated as $\mathcal{C}\otimes\mathcal{T}$. Invertedly, if we use the variable-first approach (A: 1 3 5, B: 2 4 6), the mask can be proved to be $\mathcal{T}\otimes\mathcal{C}$.

Q2: Comparison with Moirai in the zero-shot scenario.

Thanks for your valuable suggestions. We provided a comprehensive zero-shot evaluation with more large time series models including different sizes of Moirai. Please refer to $\underline{\text{Table 6}}$ in the revision for the details.

[1] Zaheer et al. Deep Sets.

Dear Reviewer V4mu,

Thanks again for your valuable and constructive review. After providing sufficient evidence and faithfully adding the requested analysis, two of the reviewers have raised the score. However, we find that the overall rating is still on the marginal threshold. We kindly hope you can consider our response and the conversation with other reviewers. If there are any further comments from your side, we will be happy to address them before the rebuttal period ends. If there are none, then we would appreciate it if you could reconsider your rating.

Thank you once again for your support and constructive feedback.

All the best,

Authors

Dear Reviewer V4mu,

We sincerely appreciate your valuable review and feedback. We are also glad to hear that our clarifications and additional experiments have addressed your concerns.

Given the borderline decision, we would like to kindly inquire about what additional information or changes would be necessary to further enhance the overall quality of our paper.

During the discussion phase, we followed the constructive comments of all reviewers, which is reflected in the revised version of 24 pages and new 100+ experiments, including:

• Empirical studies of the long-context forecasting: Given that the long-context bottleneck received less attention in the community, we reveal the performance degradation of prevalent models on well-acknowledged datasets (Figure 3) and real-world applications (Table 13) with in-depth representation analysis (Figure 11). The explorations keep to the principles of forecasting and highlight the potential of Generative Transformers, which may provide a distinct architectural choice for future deep time series models.
• A better illustration of our technical novelty: Based on the underexploited causal implementation of multivariate forecasting on Transformers, we provide an intuitive example (Figure 12) in the revision to highlight the challenges of capturing dependencies between grouped patch tokens. We revised the overall writing of the paper and better clarified the disentangling perspective underlying TimeAttention and its properties adept to multivariate time series.
• Extensive baseline models and comprehensive benchmarks: In addition to the zero-shot evaluation based on your valuable suggestion, we provide comprehensive evaluations (please refer to Q1 of Reviewer MkP7) with state-of-the-art non-Transformer models, covariate-informed models, and LLM-based forecasters, further validating that Timer-XL can be a unified backbone for different forecasting tasks.

If there are any further comments from your side, we are open to extending our evaluation and paper even further. Thank you once again for your support and the productive discussion.

All the best,

Authors

Q3: Reclarification of the contribution of extending univariate to multivariate forecasting.

Although the formulation to generalize from 1D sequences to 2D time series is straightforward, there are several unsolved challenges before our work:

• Capture Cross-Variable Dependencies: Different from previous works that regard the whole variable as a token (iTransformer and TimeXer), Timer-XL captures fine-grained dependencies between all variables in the patch level. Thus, variable correlations can be reflected by the fine-grained attention map. For example, we show the correlations of the Traffic variables that correspond to the aggregated attention map in $\underline{\text{Figure 7}}$. The fine-grained attention map can also reflect the ACF and time-unaligned dependencies.
• Maintain Temporal Causality: Timer-XL is built on decoder-only architecture, an underexploited backbone among current time series models. While encoder-only models are prevalently adopted, our empirical results reveal that keeping the causality alleviates long-context degradation better ($\underline{\text{Figure 3}}$) and boosts the performance ($\underline{\text{Table 2-5}}$). Ablation study ($\underline{\text{Table 12}}$) that analyzes different positional embeddings is also a pioneer work.
• Permutation-Equivariance of Variables: To extend univariate to multivariate data, it is crucial to ensure the positional equivalence of variables. Formally, $\operatorname{Attn}(\mathcal{P}(V1,...,VN)) = \mathcal{P}(\operatorname{Attn}(V1, ..., VN))$, where $\mathcal{P}$ is any permutation of $N$ variables. Therefore, TimeAttention is not a trivial utilization of RoPE but also satisfies the desired property: $u$ and $v$ are adopted to distinguish diagonal elements (endogenous variables) from non-diagonal elements (exogenous variables). In this way, the permutation-equivariance of variables can be satisfied (this work [5] provides detailed proof).

To our best knowledge, the paradigm of next token prediction is still unexplored in task-specific forecasters, let alone extend to multiple forecasting tasks. Without changing the self-attention, we design a masking mechanism to disentangle the large attention map into formalizable and interpretable temporal and variable dependencies. In this perspective, our implementation is simple but effective. In summary, here is a comparison with previous time-series Transformers:

Q4: Motivations of enlarging the context length.

We agree with your opinion that long contexts would not always be beneficial. In fact, we want to draw the attention of the community to the long-context bottleneck instead of advocating for everyone to adopt long contexts. In addition to $\underline{\text{Figure 3}}$, we provided a new case study here :

In the meteorological domain, it is of great demand to support the context of more than years to contain a specific cycle (such as El Nino). More importantly, it draws similar conclusions that encoder-only Transformers encounter performance degradation in long context series (even inferior to the linear model at last), which can be concealed due to the short context adopted in previous benchmarks. Although Timer-XL with the longest context is still worse than using a searched one in $\underline{\text{Figure 3}}$, it is more a matter of feature engineering. We focus more on the architecture, with the results revealing Generative Transformers can better cope with long contexts. By our visualization of attention in encoder-/decoder-only Transformers ($\underline{\text{Figure 11}}$ in the revised version), the decoder-only model can selectively focus on long contexts, while attention maps of PatchTST are oversmooth and make the model wrongly focus on the noisy part.

Thank you again for your comments, which are very helpful for us to improve the quality of the paper. We provided comprehensive benchmarks and clarified more on the technical design and long-context bottlenecks. Please reconsider the efforts and contributions of our work. If you have any further questions, we are looking forward to discussing with you.

Another inspiring fact is that Generative Transformers have flexibility on the input length (as long as the input length during inference is shorter than the context length during training). It leaves a promising application: Training a model with a long context and applying it with only partial length or being adapted to a specific length.

Therefore, long context is a basic model capability towards foundation models, which can support emergence capabilities such as prompting, ICL, RAG, etc. However, the long-context forecasting paradigm received less attention in the current community, which can be due to the lack of benchmarks. Therefore, we spent a lot of time curating the ERA5 dataset (500 thousand time points with yearly context and thousands of variables), which serves as a contribution that cannot be ignored.

We appreciate your suggestion about using a large patch size, which can shorten the context length and reduce steps of autoregression. We provided the results using a large patch size in the middle of $\underline{\text{Figure 8}}$. The finding is noteworthy and indicates potential improvement in future works.

Q5: Illustration of the patch token for univariate and multivariate data.

Thanks for your valuable suggestions. For a better illustration, we provided $\underline{\text{Figure 12}}$ in the revision, which highlighted the challenges and our pipeline of capturing fine-grained dependencies between patch tokens.

[1] Time-LLM: Time Series Forecasting by Reprogramming Large Language Models. ICLR 2024.

[2] Large Language Models Are Zero-Shot Time Series Forecasters. NeurIPS 2023.

[3] FITS: Modeling Time Series with 10k Parameters. ICLR 2024.

[4] TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. ICLR 2024.

[5] Zaheer et al. Deep Sets.

Many thanks to Reviewer MkP7 for providing a valuable review.

Q1: Comparison with LLM-based and non-Transformers forecasters.

Thanks for your valuable suggestions, we extensively included state-of-the-art methods to enlarge our forecasting baselines in $\underline{\text{Table 3}}$: Time-LLM[1], LLMTime[2], TimeMixer[3], and FiTS[4]. Here are the results:

Overall, our model achieves the best performance under 78% settings.

Q2: More discussion of the computational cost of Timer-XL.

Thanks for your scientific rigor. We included quantitive results of the computational cost on high-dimensional time series (ECL: 321 variables) with a long lookback length (672), leading to the context of 200k+ time points. We use the same model hyperparameters according to the configurations of $\underline{\text{Table 10}}$:

Results are consistent with our theoretical analysis: Under the same hyperparameters, parameter counts are almost the same, while the multiplier of flops is less than the number of variables ($N$). We will also take it as essential work to improve the model efficiency of channel-dependent models.

Thank you again for your detailed comments, which are very helpful for us to improve the quality of the paper. We clarified the positioning and contribution more rigorously, addressed the concerns, and enhanced the comprehensiveness of experiments. Please reconsider our contributions and insights from our work. If you have any further questions, we are looking forward to discussing with you.

• Experiment 4.3 - Covariate-Informed Forecasting: We reported the baseline results from TimeXer, where PatchTST and other models are adapted to incorporate covariates. Also, we added new task-specific forecasters:

  MSE|MAE	DeepAR	N-BEATSX	Timer-XL
  NP	0.277 | 0.308	0.272 | 0.301	0.234 | 0.262
  PJM	0.121 | 0.213	0.097 | 0.189	0.089 | 0.187
  BE	0.416 | 0.279	0.389 | 0.265	0.371 | 0.243
  FR	0.432 | 0.237	0.393 | 0.211	0.381 | 0.204
  DE	0.512 | 0.458	0.499 | 0.447	0.434 | 0.415

• Experiment 4.4 - Zero-shot forecasting: Since the benchmark of zero-shot forecasting is under rapid development. We provided zero-shot results on these datasets that are not included in UTSD during pre-training.

  Zero-shot (MSE|MAE)	Timer-XL	Timer
  BDG-2 Panther [5]	0.331 | 0.370	0.354 | 0.395
  Cockatoo [5]	0.278 | 0.377	0.324 | 0.396
  GFC17 Load [5]	0.289 | 0.377	0.340 | 0.425
  Taxi New York Passengers [6]	0.179 | 0.302	0.524 | 0.580
  Temps [6]	0.790 | 0.702	0.981 | 0.782

  Please feel free to let us know if you want us to evaluate the performance on more datasets. We also provided zero-shot evaluation with more large models. Please also refer to $\underline{\text{Table 6}}$ in our revision for the details.

Q5: Support for probabilistic forecasting.

It is necessary to support probabilistic forecasting because many task-specific models lack this capability. It is a direction worthy of specific study. We will explore the following direction: integrating proper distribution of time series and employing probabilistic prediction heads for heterogeneous variables. Your suggestion will guide us in refining our methodology.

[1] UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecasting.

[2] In-Context Fine-Tuning for Time Series Foundation Models.

[3] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts.

[4] Interpretable Weather Forecasting for Worldwide Stations with A Unified Deep Model.

[5] Unified Training of Universal Time Series Forecasting Transformers.

[6] A Decoder-Only Foundation Model for Time Series Forecasting.

Q3: Reclarification of the technical novelty of our work.

Thanks for your valuable feedback regarding the novelty. Our contribution lies in two aspects:

1. Generative Transformers for Long-Context Series: As you mentioned, long-context Transformers for multivariate modeling have been studied. Moirai and UniTST are all encoder-only models. Our difference lies in the implementation of Generative Transformers. Motivated by the success of GPTs in NLP, which successfully accommodates millions of tokens, our work validated that they can better solve performance degradation in long-context time series.

To our best knowledge, Generative Transformers are still underexploited among task-specific models, let alone the implementation for multivariate series. In this perspective, our technical novelty is not limited and does fill in the blanks.

Although the longest context may be inferior to a searched one ($\underline{\text{Figure 3}}$), it is more a matter of feature engineering, and searching also requires heavy time and resources. A longer lookback does introduce irrelevant noise and distribution shift. Therefore, finding an architecture that can adaptively select information in vast observations is significant, which highlights our contribution of adopting Generative Transformers on long-context univariate forecasting.

2. TimeAttention for Unified Forecasting: The equation of calculating attention is inspired by Moirai. We greatly appreciate the pivot work with the explicit citation. However, we introduce the masking formulation, whose key lies in the grouped causality of flattened 2D sequences. We derive it based on the Kronecker Product, which can disentangle the large attention map (as shown in $\underline{\text{Figure 7}}$) into explicit temporal and variable dependencies. It can generalize to pre-defined correlations (such as covariates), which is unexplored in previous works.

To sum up, we explore the potential of Generative Transformers for (1) long-context univariate series and (2) multiple time series. Technically, we provide a decoupling perspective of multivariate next token prediction and implement it on the decoder-only architecture, which may inspire future explorations.

Q4: Further explanations of experiments in our work.

We appreciate your questions regarding the experiments. We actually spend a lot of time on these experiments. Please allow us to clarify the experiments you mentioned in detail:

• Experiment 4.1 - Long-Context Univairtae Forecasting: Even though we only compared two models here, they are representative and work exactly for our studied problem: Which architecture of Transformer can better capture temporally long dependencies? In this univariate setting, Timer-XL is degraded into the decoder-only Timer, and the corresponding counterpart is encoder-only PatchTST (UniTST/iTransformer will be regarded as PatchTST/FFN layers). Further, we added DLinear based on your inspiration in the revised $\underline{\text{Figure 3}}$. For the fairness of comparison, please refer to $\underline{\text{Table 10}}$, where we keep the hyperparameters of Transformers all the same.

• (Minor): Section 4.1 discusses instance normalization (ReVIN). Since ReVIN can generally improve the performance of the previous encoder-only Transformers but leads to special problems in Generative Transformers (e.g., unmatched statistics in multi-step autoregression), we provide more results indicating that using Timer-XL without ReVIN can still achieve competitive performance in these well-acknowledged benchmarks, while the performance of PatchTST may heavily rely on this normalization:

  Dataset-672-Pred-96 (MSE|MAE)	Timer-XL with ReVIN	Timer-XL w/o ReVIN	PatchTST with ReVIN	PatchTST w/o ReVIN
  ETTh1	0.364 | 0.397	0.370 | 0.401	0.370 | 0.399	0.421 | 0.448
  Weather	0.157 | 0.205	0.151 | 0.205	0.149 | 0.198	0.173 | 0.242
  ECL	0.127 | 0.219	0.130 | 0.225	0.129 | 0.222	0.138 | 0.244

• Experiment 4.2 - Multi-Station Forecasting: Since it takes a long time to train on GTWSF, we report the original results of baseline models from Corrformer[4]. According to your suggestion, we added UniTST to this benchmark and revised. $\underline{\text{Figure 4}}$.

  MSE	UniTST	Timer-XL
  Global Temp	7.356	7.172
  Global Wind	3.807	3.786

• (Minor): Thanks very much for pointing out the wrong labels! We fixed it in $\underline{\text{Figure 4}}$.

Many thanks to Reviewer r6jB for providing a detailed and in-depth review, which helped us significantly improve the quality of our submission.

Q1: Clarification of the position of our work.

Thank you very much for providing such an essential suggestion. Our work aims for a unified backbone for one-for-all forecasting. Our evaluations are divided into two parts: supervised training on Timer-XL for different tasks (Sections 4.1, 4.2, and 4.3) and large-scale pre-training on long-context time series (Section 4.4).

We apologize for the potential misleading caused by several large models in the supervised training baseline, such as Moirai. Except for Section 4.4, Timer-XL, Moirai, and other baseline models are trained from scratch. Therefore, we revised the name of Moirai to the more appropriate UniTST[1] (w/o dispatchers) in these sections, a pioneer encoder-only backbone supporting unified forecasting. In Section 4.4, we intend to validate that long-context pre-training empowers large time series models (we appreciate your suggestion to use more comprehensive benchmarks, which we resolved in $\underline{\text{the response to Q4}}$).

Your suggestion helped us better clarify the scope of our work, and we revised the overall writing of the paper.

Q2: Empirical justification to reveal the context bottleneck.

We agree with your opinion that enlarging contexts would not always be beneficial, which needs a better empirical justification. As per your request, we provided more practical cases of long-context forecasting where using long contexts is an essential requirement: In the meteorological ERA5, despite its low sampling frequency (3 hours), it is necessary to support the context of more than years to contain a specific cycle (such as El Nino):

The performance of Timer-XL and DLinear generally improves with the increased context length. It also reveals the performance degradation of PatchTST. Similar to the observations in $\underline{\text{Figure 3}}$, encoder-only Transformers encounter the degradation earlier, which can be concealed due to the short context adopted on previous benchmarks.

Although PatchTST has conducted an initial exploration in the context of hundreds of time points, it is inferior in ever-long contexts, and we find that Generative Transformers mitigate performance degradation well. We further analyze this problem:

• Increased Supervision: Generative Transformers predict all tokens, including tokens in the lookback window.
• Causality Maintainance: Using causal masks leads to coherent token embeddings, while the unmasked attention in PatchTST breaks the causality and cannot tell these tokens.

We provided representation analysis of encoder-/decoder-only Transformers in $\underline{\text{Figure 11}}$ of the revised version, revealing that the decoder-only model can selectively focus on the long context while PatchTST may wrongly focus on noisy parts.

In addition to the empirical justification, we notice that recent large time series models also paid special attention to longer context length[2][3]. This capability serves as the basis for supporting emergent capabilities such as prompting, ICL, RAG, etc. Therefore, we believe that context bottlenecks deserve further exploration in this community.