FlexTSF: A universal forecasting model for time series with variable regularities

1. Some notations need further clarification. For example, the notation $(x_i, t_i)$ in Line 133, do the two subscripts $i$ share the same meaning? The notation $\Delta t$ seems been multi-defined in Line 187, Line 211, and Line 229.
2. Some highly relevant related works are missing, such as iTransformer for regular time series data and CRU for irregular time series data. [1] Liu, Yong, et al. "itransformer: Inverted transformers are effective for time series forecasting." ICLR 2023. [2] Schirmer, Mona, et al. "Modeling irregular time series with continuous recurrent units." ICML 2022.
3. The paper does not provide a detailed experimental setup, especially the forecast horizon. It also lacks a specific explanation for why the sequence length is set to 192 for ETTh2 and 512 for ETTm2 and why the held-out dataset does not include ETTh1, ETTm1, ECL, and Traffic datasets.

Experimental Setup Issues

• The article does not provide a detailed description of the experimental setup, including input and output lengths, which significantly affects the results. This lack of information makes it challenging for reviewers to compare the results intuitively with previous works in temporal prediction, ultimately diminishing the credibility of the model's effectiveness.
• Additionally, the paper's baselines do not include recent significant contributions in temporal foundational models, such as Timer[1], Moment[2], TTM[3], and Time-MOE[4].

The novelty of the article is somewhat limited.

• VT-norm:
  • The two-stage normalization essentially combines overall normalization and instance normalization, which are common techniques in time series processing.
  • The authors concatenate all features into a vector to feed into the model as a prompt, claiming this approach addresses domain diversity. However, this method has been utilized in various studies, such as TimeLLM[5], which encodes statistical and domain information into prompts for LLMs. Moreover, relying solely on statistical information from the dataset does not effectively resolve the issue of domain diversity, as different domains may share similar statistical properties.
• LED Attention:
  • Leader Node: The Leader Node can be viewed as a prompt based on statistical information from the dataset, which has also been used in many other models.
  • Dummy Patch: This concept is similar to the decoder inputs in previous models like Informer[6] and Autoformer[7].
  • Positional Encoding: The approach is simply a straightforward application of RoPE.

Reference

[1] Liu, Y., Zhang, H., Li, C., Huang, X., Wang, J., & Long, M. (2024). Timer: Transformers for time series analysis at scale. arXiv preprint arXiv:2402.02368.

[2] Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., & Dubrawski, A. (2024). Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885.

[3] Ekambaram, V., Jati, A., Nguyen, N. H., Dayama, P., Reddy, C., Gifford, W. M., & Kalagnanam, J. (2024). TTMs: Fast Multi-level Tiny Time Mixers for Improved Zero-shot and Few-shot Forecasting of Multivariate Time Series. arXiv preprint arXiv:2401.03955.

[4] Shi, X., Wang, S., Nie, Y., Li, D., Ye, Z., Wen, Q., & Jin, M. (2024). Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts. arXiv preprint arXiv:2409.16040.

[5] Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., ... & Wen, Q. (2023). Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728.

[6] Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34, 22419-22430.

[7] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021, May). Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 12, pp. 11106-11115).

I have several critical concerns, which are summarized below.

1. Seemingly Complicated Designs, Yet Essentially Combinations of Existing Techniques

This paper introduces several new technical names that are essentially derived from existing studies. For instance, VT-Norm employs a global preprocessing method for time-series values, followed by per-batch instance normalization, and also normalizes timestamps. IVP Patcher utilizes a technique from another study that solves initial value problems (IVP) to compute the patch embedding. LED Attention is essentially causal self-attention, but with some additional "tokens" as inputs, raising the question of why a new name for attention is necessary. The VAE-style training is akin to existing studies like mTAN. I suggest that the authors include techniques from existing studies in a preliminary section and focus on their unique designs and adaptations in the methodology section.

2. The Rationale of Using Irregular Time Series to Boost Regular Forecasting

I understand that traditional models for regular time series do not perform well on irregular time series due to sampling irregularities. However, it's unclear how irregular time series and the derived architectures improve performance on regular time series. For instance, Table 5 is confusing to me because pre-trained time-series models significantly underperformed compared to FlexTSF on regular time-series benchmarks. Is this due to differences in pre-training data or the unique design of FlexTSF? Unfortunately, I don't find a specific explanation for these performance differences, leading me to question the robustness of the experimental process.

3. Lack of Verifications on Irregular Time-series Classification and Comparisons with Other Available Baselines

In the context of pre-training irregular time series, it's crucial to verify the representation learning capability of FlexTSF on typical irregular time-series classification tasks. Most irregular time-series models in the literature are evaluated based on their ability to learn effective representations for classification tasks, particularly in healthcare scenarios. For example, irregular time-series baselines such as GRU-D, mTAND, Latent-ODE, and ContiFormer follow this approach. In this thread, more recent studies, such as Warpformer [1], are not included.

Addtionally, a published study focusing on irregular time-series forecasting [2] provides a comprehensive comparison of various models applicable to irregular time-series forecasting.

[1] Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Series, https://dl.acm.org/doi/10.1145/3580305.3599543

[2] Irregular Multivariate Time Series Forecasting:A Transformable Patching Graph Neural Networks Approach, https://openreview.net/pdf?id=UZlMXUGI6e

• Although FlexTSF is evaluated against state-of-the-art baselines, the comparison could be extended to include recent foundational models in time series, such as UniTS (NeurIPS 2024) and Reprogrammed LLMs for time series. Inclusion of these models would provide a more comprehensive benchmark. There are forecasting models missing from the comparison, such as DeepAR for probabilistic forecasting and Autoformer for long-sequence time series forecasting.

• The paper centers exclusively on forecasting as the primary task. Expanding FlexTSF to support other time series tasks—such as classification, anomaly detection, and imputation—could broaden its usability. These tasks are highly relevant in domains where time series models often need to operate on diverse tasks, and demonstrating FlexTSF across time series tasks would make the model more compelling as a foundation model.

• A combination of IVP Patcher and LED Attention make interpreting model outputs challenges. Additionally, using neural IVP solvers and LED Attention layers can be computationally intensive. It would be helpful to profile the computational load FlexTSF.

• FlexTSF’s approach does not explicitly address and is tested under forecasting with variable horizon lengths, where the desired prediction window may change dynamically based on the application. This could limit the model’s applicability in settings where adaptive forecasting horizons are needed, such as in finance and emergency response.