Toto: Time Series Optimized Transformer for Observability

• The paper proposes another pretrained model for time series such as Chronos, TimesFM and Moirai. When compared with these existing models, the technical contribution of Toto is marginal. The mixture of distributions idea is not new and has been studied in forecasting literature before, most recently even in the context of pretrained models (Moirai). Furthermore, the alternating spatial and temporal attention blocks idea is incremental.

• In the absence of strong methodological contributions, the empirical investigation needs to be strong to meet the acceptance bar. Unfortunately, the experiments reported in this paper are sub par. Concretely, they have the following limitations.

  • The breadth of experiments is severely limited for a purported "foundation model". In the benchmark on publicly available datasets (Table 1), only 6 datasets have been studied, 5 of which belong to the same domain with 4 being essentially the same dataset (ETTh1, ETTh2, ETTm1, ETTm2). This benchmark is limited even for a regular time series paper. For a pretrained model, this is not sufficient to draw any conclusions. Please check the Chronos and/or TimesFM papers for a better evaluation benchmark.
  • Even on these 6 datasets, it appears that the numbers reported for Toto are not from the same setting as other methods. The numbers for baselines such as Moirai and TimesFM have been reproduced from their papers but the rolling evaluation for Toto appears to have been conducted with a stride of 512 which is not the same setting as in the baselines. Due to the fewer slices being evaluated, the numbers could be inflated. Please clarify if I have misunderstood the setting. See questions for details.
  • The second benchmark for observability time series appears to have been conducted on a proprietary dataset which makes it difficult to verify for the community. Will the dataset or the pretrained model weights be released? In either case, code/model could have been made available to the reviewers. Regardless of reproducibility concerns, the percentage metrics reported are often discouraged in the literature (see [1] for details). Furthermore, it is unclear what the key claim of this experiment is. Since Toto has been trained on large amounts of data from the same observability domain (75% of 1T time series), it's not surprising that it would perform better than other "zero-shot" models.
  • Beyond reporting the numbers, the paper does not conduct any serious investigation of the models or the results.

• Minor: It's not clear to me what a "foundation model" for time series forecasting in observability really means. With such domain-specific qualifier, the word "foundation" loses its meaning and it would be better replaced with something like "pretrained".

[1] Hyndman and Koehler (2006). Another look at measures of forecast accuracy.

The two main weaknesses of this manuscript are that the architectural contributions are incremental and the validation on the long sequence forecasting benchmark requires either revision or clarification.

Incremental contribution: I would argue that the main contribution of this paper is the observability dataset. A high quality dataset for this domain would be helpful to further develop foundation models for observability specifically and for time series foundation models more generally. However, since the data in this work is proprietary this will not benefit the ICLR or larger science community. The other contributions (proportional factorized space-time attention) and the mixture student-t probabilistic output distribution are incremental. The proposed transformer architecture is a minor modification from Arnab et al. ICCV 2021 (https://arxiv.org/abs/2103.15691) with a configurable parameter to adjust the ratio of space and time attention layers. The mixture student-t distribution is a minor modification of the mixture distribution output in Woo et al. ICML 2024 (https://arxiv.org/abs/2402.02592). GluonTS (Alexandrov et al., JMLR 2020) also provides an implementation of a generic mixture distribution that can be instantiated as a student-t mixture.

Thus, I would regard the contributions as too incremental to warrant publication in ICLR.

Evaluation on Long Sequence Forecasting (LSF) Dataset: Another concern is on the evaluation on the LSF dataset. The setup in this manuscript uses a rolling evaluation with a stride 512, which is inconsistent with the results taken from earlier work that is included in this evaluation (Woo et al. ICML 2024). Specifically, the original setup of this dataset uses a stride of 1 (Zhou et al., AAAI 2021: https://arxiv.org/abs/2012.07436). Hence, the results of Toto and the other results that are taken from earlier manuscripts are not comparable.

I would kindly ask the reviewer to perform the experiments under the same condition as the manuscripts where the experimental results are reported from. I will consider raising my score if the experiments are updated or it is clarified that the experiments are indeed performed under the same conditions.

W1: The paper’s two primary contributions are the novel observability data and the newly designed foundation model. However, neither the data nor the model are publicly available, raising concerns about reproducibility and broader applicability.

W2: The technical novelty is limited; for instance, the probabilistic forecasting using the Student-T mixture model (SMM) is an extension, as the Student-T distribution as a prediction head has already been proposed [1].

W3: In Table 1, the baseline results are cited directly from Woo et al. (2024) and Das et al. (2024). However, the current information provided does not allow for adequate evaluation.

W4: The experiments lack essential details, such as the historical context length in Section 4.1, the sampling strategy and quantity, and the configurations for historical context length and prediction length in Section 4.2.

W5: The experimental evaluation is insufficient, omitting several important baselines. Additionally, the proposed method lacks a parameter sensitivity analysis.

• The work lacks clear motivation; the authors’ objective is difficult to discern.
  • If the aim is a superior observability model, why opt for a foundation model? Any deep forecasting backbone could be trained on similar data for comparison. Instead, only zero-shot models are evaluated on the observability benchmark, where presumably the proposed model was pre-trained.
  • If the goal is to demonstrate generalization of the model trained on observability data to other domains, more comprehensive benchmarks are necessary (e.g., Chronos benchmark; LSF is overly simplistic).
  • If the focus is on the efficiency of the factorized space-time attention mechanism, standard task-specific training regimes would suffice instead of pre-training and zero-shot evaluation.
• If domain-specific training data is a contribution, details are needed on the pre-training corpus construction, ablation studies, and novelty of data selection.
• The paper’s presentation could be significantly improved in terms of completeness, clarity, and structure.
• The technical contributions of space-time attention and Student-t mixture heads are of limited novelty and provide marginal improvement.
  • Experimental results are insufficient to substantiate the claimed contributions:
  • There is no empirical evidence of the computational efficiency of the proposed attention module.
  • For the observability benchmark, only zero-shot model results are presented. Given the extensive pre-training on observability data, fine-tuned versions of the baseline models would provide a fairer comparison. Additional baselines (e.g., classical models, task-specific deep learning models such as TFT, TiDE) should be included.