TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learning

We thank the reviewer for their thorough summary and for acknowledging the strengths of our work, including the novel integration of xLSTM, the benefits of CPM, and the model's state-of-the-art performance and efficiency. In the following, we address the reviewer's views on weakness as requested in the questions by the reviewer:

1.Currently focuses on univariate time series analysis, with multivariate modeling explicitly noted as a direction for future research.

The reviewer is correct that TiRex models each time series variate independently. We identified this as a limitation and an area for future work. However, we want to emphasize that this approach remains remarkably effective. Previous work (e.g., [1]) has shown that univariate models can be strong on multivariate tasks.

Our results on the GiftEval benchmark, which includes multiple multivariate datasets (8 out of 23), confirm this. TiRex outperforms models designed specifically for multivariate forecasting (e.g., Moirai, TTM) on the overall benchmark (Figure 1). To make this point more explicit, the table below shows the performance (Top 3 models by CRPS) on just the multivariate datasets within the GiftEval-ZS benchmark, where TiRex still achieves the top rank. This demonstrates that for many real-world problems, accurately modeling individual series dynamics is the most critical factor.

We will add this extended discussion to the paper.

2. Omits evaluation on widely adopted benchmark datasets (e.g., ETTH1, ETTH2, ETTM1, ETTM2, Weather, Electricity) that are routinely used in state-of-the-art (SOTA) studies. This gap limits direct comparability with prior works and may obscure the model's generalizability across diverse time series modalities.

We note that our evaluation already includes all the datasets the reviewer is concerned about. As detailed in our appendix, the GiftEval benchmark includes ETTh1, ETTh2, ETTm1, ETTm2, and Electricity. The Chronos-ZS benchmark includes the Weather dataset. The individual results are presented in Tables 9 to 18 in the appendix.

Our choice to use the GiftEval and Chronos-ZS benchmarks instead of the older LFS-benchmark (which consists of the datasets mentioned) was deliberate and made to ensure a more rigorous, transparent, and comprehensive evaluation. We will explain this in the final manuscript.

In short, recent work [e.g., 2,3] has criticized the LFS benchmark for issues with inconsistent evaluation protocols and limited diversity, which hinder generalizable comparison. In contrast, GiftEval and Chronos-ZS offer:

1. Greater Diversity and Generalizability: GiftEval contains 97 evaluation settings across 23 diverse datasets, while Chronos-ZS contains 27 datasets --- both covering diverse domains. This provides a much broader test of a model's zero-shot capabilities than the 6 datasets in the LFS benchmark.
2. Standardization and Reproducibility: Both benchmarks are hosted on public leaderboards with standardized evaluation code, ensuring that all models are compared fairly under the same conditions. This resolves comparability issues present in prior work.

Therefore, by using these comprehensive benchmarks, we are confident that our evaluation provides a more robust and comparable assessment of SOTA performance across a wide range of time series domains.

3. Lacks a publicly available codebase for independent verification, which hinders reproducibility and community validation of the reported findings.

We fully agree with the reviewer on the importance of reproducible evaluation results. We will release our model, the respective code, and the evaluation scripts to allow for the full verification of our benchmark results. The reproducibility of our findings will be also ensured by our submissions to the public GiftEval and Chronos-ZS leaderboards, which require the necessary code and model to validate the results.

We thank the reviewer for the opportunity to address their feedback. Given that we have now addressed primary concerns --- e.g, the evaluation results for the noted datasets and committing to a public release of our model --- we hope the reviewer will consider adjusting the initial score accordingly.

References

[1] Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR 2022.

[2] Bergmeir, C. Fundamental limitations of foundational forecasting models: The need for multimodality and rigorous evaluation, Time Series in the Age of Large Models Workshop NeurIPS 2024

[3] Aksu, T., Woo, G., Liu, J., Liu, X., Liu, C., Savarese, S., Xiong, C., and Sahoo, D. GIFT-eval: A benchmark for general time series forecasting model evaluation. In NeurIPS Workshop on Time Series in the Age of Large Models, 2024.

We thank the reviewer for their feedback and for positively highlighting the experimental design, the substantial performance improvements, the novelty of the architecture, and the quality of the writing. In the following, we address the reviewer's views on weaknesses and answer the questions by the reviewer, point by point:

1. There is very little theoretical analysis, it is not clear on a fundamental level why TiRex does better than other models, and what its potential failure modes are.

We appreciate the reviewer's desire for a deeper theoretical understanding. While our paper focuses on the construction of the approach and its empirical validation with the respective ablations, we believe one can ground TiRex's superior performance at least partly in the established theoretical properties of its backbone: As shown by [1,2,3], recurrent architectures like sLSTMs occupy a higher class of formal languages than transformers. Specifically, their state-tracking ability allows them to solve problems (e.g., parity, counting) that are provably beyond the reach of standard transformers (Figure 7). We hypothesize that this fundamental expressivity advantage allows TiRex to better model temporal dynamics, leading to better performance, especially across long horizons. We alluded to this aspect in the appendix, but the reviewer comment shows that we did not link this well enough. Hence, we will clarify this connection in the final version. Regarding failure modes, we address this in the second next point.

1. The authors showed that TiRex can predict periodic spikes well, which is a common failure mode for other time-series foundation models. This is not surprising given the specific data augmentation techniques (e.g., spike injection) employed to train TiRex. But how well can TiRex generalize to situations not well covered in the training data. For example, what will TiRex predict when faced with aperiodic spike trains (e.g., from neuroscience)? It would be interesting to test TiRex on synthetic time series where different aspects of the signal can be controlled.

While our spike injection augmentation was indeed key, we designed it specifically to encourage generalization rather than memorization:

1. Diverse Spike Features: The augmentation is more complex than simply adding a repeating spike. As detailed in Section 3 and Appendix B, we introduce significant diversity by randomizing:

• Spike Shape: We use multiple kernels (Tophat, RBF, Linear) to define the spike's shape.
• Spike Parameters: The width and amplitude of each spike are sampled from distributions.
• Temporal Structure: The periodicity itself is randomized for each sample, and we use a variety of underlying temporal patterns. The goal of this diverse training is to teach the model a general concept of "sharp, transient events" and understand them “in-context”, rather than having it memorize a simple periodic pattern.

We will extend the explanations in the main paper (Section 3) and Appendix B to make this more clear.

2. Predictability of Aperiodic Spikes: A truly stochastic spike, with no causal link to the preceding signal, is by definition unpredictable from the context alone. In this case TiRex might provide an estimation of the expectation with high uncertainties. However, if spikes are triggered by the system's underlying dynamics, we hypothesize TiRex would be uniquely suited to generalize as the sLSTM's state-tracking, might allow to model the latent state leading up to an event - a potential advantage over architectures that may be limited to pattern matching.

2. What are some failure modes of TiRex that the authors have observed? Understanding these failure modes will help further improve the model.

While we did not observe any specific repetitive failure modes in the evaluation, two hypothetical ones are worth noting:

• As discussed in the limitations section, our current model treats multivariate time series as a collection of univariate ones. While this approach is strong, as reflected in the Gift-Eval results (8 out of 23 datasets are multivariate) and pointed out in previous work [4] it may fail in scenarios where forecasting is impossible without explicitly modeling interdependencies between variates.
• As a zero-shot, in-context learning model, TiRex's performance depends on having a sufficient context of past values to identify the time series' properties. When faced with a very short context that does not exhibit the characteristic patterns of the series, its forecasts may be unsatisfactory, a natural limitation of this paradigm.

3. Did the authors find any interpretable strategies TiRex is using to make zero-shot forecasts?

We did not find any specific interpretable strategies. However, we also did not explicitly search for them. It is likely that for specific forms of time series TiRex would use straightforward strategies to construct its forecast.

References

[1] Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. xLSTM: Extended Long Short-Term Memory. NeurIPS 2024.

[2] Merrill, W. and Sabharwal, A. The parallelism tradeoff: Limitations of log-precision transformers. TACL 2023

[3] Delétang, G., Ruoss, A., Grau-Moya, J., Genewein, T., Wenliang, L. K., Catt, E., Cundy, C., Hutter, M., Legg, S., Veness, J., and Ortega, P. A. Neural networks and the Chomsky hierarchy. ICLR 2023.

[4] Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. ICLR 2022.

We thank the reviewer for their careful summary of our work. We are glad they recognized the novelty of leveraging the xLSTM architecture and our proposed Contiguous Patch Masking (CPM) training strategy. We especially appreciate that the reviewer highlighted the impact of CPM in addressing the collapsing quantiles problem and its significant contribution to the model's overall performance.

We thank the reviewer for their feedback and for highlighting the significance of TiRex's performance improvements, its architectural novelty, the thoroughness of the evaluation, and the practical utility of the ablation studies. In the following, we address the reviewer's views on weaknesses and answer the questions by the reviewer, point by point:

1. It is a improvement over the xLSTM architecture with additional novelty only in input patching and simple data augmentation

While the xLSTM backbone is indeed a foundational element we would like to clarify our contributions, as our work introduces two other key contributions that are crucial for achieving the final performance.

• Contiguous Patch Masking (CPM): CPM combined with our “masked inference “mechanism addresses the challenge of autoregressive error accumulation in long-horizon forecasting, enabling stable and coherent predictions where previous models fail. This is in contrast to a naive multi-patch training approach, which we show harms short-term performance, and standard autoregressive inference, which degrades long-term performance (Table 1).
• Data Augmentations: The proposed online augmentations demonstrate their importance in improving the model's robustness and generalization capabilities (Table 1).

The synergistic combination of the xLSTM architecture, the CPM training strategy, and our specialized data augmentations, which together establish the TiRex model, therefore represent our main contributions.

1. Can the model be used for other tasks like imputation, anomaly detection and classification?

While we did not analyze these tasks in the paper, we believe the model holds significant potential for them. Similar to how pre-trained models in other domains provide powerful feature extractors, the hidden representations of TiRex could be leveraged for various downstream tasks. The state-tracking capability of the sLSTM modules, in particular, may provide a rich, compressed representation of the time series dynamics, suitable for tasks like anomaly detection or classification. We will add this discussion to our future work section.

2. How does the performance vary with horizon length and frequency of time-series?

Our evaluation shows that TiRex excels across various horizon lengths and frequencies.

• Horizon Length: Figures 9, 10, and 11 in the appendix provide a detailed performance breakdown on the GiftEval-ZS benchmark for long-term (600-900 steps), medium-term (100-600 steps), and short-term (< 100 steps) forecasts, respectively. TiRex achieves the top rank in all three categories, with its most significant performance gains observed in long- and medium-term forecasting.
• Frequency: The GiftEval and Chronos-ZS benchmarks were chosen specifically for their diversity, hence also covering different frequencies (as detailed in Tables 7 and 8). TiRex's number one overall rank on both benchmarks (Fig 4 and 5) demonstrates its strong and robust performance across these different time scales. For GiftEval TiRex is the top-performing model for daily, hourly, minutely, and secondly datasets, and among the top three for weekly, monthly, and quarterly datasets.

3. Can the hidden states be interpreted to model different dependencies and explanations similar to attention modules?

While a detailed interpretability study was beyond the scope of this work, the architecture of TiRex is promising for such analysis. Unlike transformer models, which rely on attention maps for interpretation, TiRex utilizes sLSTM modules with explicit recurrent state vectors. We hypothesize that this state vector acts as a compressed summary of the time series history. Analyzing these state dynamics could offer a powerful and potentially more direct way to understand the model's forecasting strategy, in future research.