This Time is Different: An Observability Perspective on Time Series Foundation Models

Thank you for your review and thoughtful questions.

The primary module proposed in the method section has been used in other Foundation Models, such as variable association modeling and Student-T mixture model Head.

MoIRAI also employs a Student-T Mixture for probability modeling, but the Student-T Mixture can only fit specific distributions. Please clarify how this differs from TOTO.

The UNiTS model also utilizes Attention for variable correlation modeling. Please explain the differences compared to your approach.

While we agree there are similarities between Toto’s architecture and other recent works, let us attempt to clarify which contributions we feel are meaningful and novel:

Student-T Mixture Model (SMM)
While other authors have used mixture models (e.g. Moirai), to our knowledge Toto is the first time series foundation model to use a Student-T Mixture Model head.

We cite the prior art from Moirai, and took it into consideration when designing our mixture head; however the two approaches differ in crucial and meaningful ways. Moirai does not use a Student-T mixture model. As stated in their paper, Moirai uses a fixed mixture of four different distributions [1]. The Moirai authors suggest that these four individual components were chosen heuristically to match specific common cases within time series datasets.

By contrast, Toto's mixture consists of only Student-T distributions, where the number of components, $k$, is a hyperparameter that we optimize ($k$=24 in our selected model). This is fundamentally a more general and less heuristic approach. We derive it from first principles: mixtures of certain distributions (including Gaussians and Student-T’s) are universal approximators for any probability density function when using a sufficiently large number of mixture components [2, 3]. We choose the SMM (rather than GMM for example) based on its superior performance with heavy-tailed distributions [4], which frequently appear in the observability domain.

We also adopt a novel loss formulation which leads to significantly better convergence when training with our mixture model head. Alongside NLL, we jointly optimize the SMM with Robust Loss [4] to concentrate the mean towards strong point predictions.

Based on this feedback, we will revise the discussion in our Related Works section to make these distinctions clearer.

Causal patch-based instance normalization
This is a novel form of normalization which dramatically improves performance in our ablation study (Figure 3B). It does this by preserving the causality of our autoregressive decoder-style model, which is critical for highly nonstationary time series (where future information leakage can cause severe overfitting).

Proportional factorized attention
In Section 3.1, we cite previous demonstrations of factorized attention [5-7]. Our novel contribution introduces an asymmetrical ratio of layers, with the ratio of timewise to variate-wise attention operations as a tunable hyperparameter. While this is a relatively simple idea, it represents a meaningful improvement in computational efficiency. Our design is motivated by the observation that channel-independent models such as TimesFM [8] and Chronos [9] still achieve competitive performance on multivariate benchmarks and, therefore, we expect diminishing returns on FLOPs spent on variate-wise attention operations.

We will add a citation and discussion of UNiTS to make this distinction clearer.

The experiment is imperfect, lacking model analysis experiments, ablation experiments, and parameter sensitivity analysis.

We do perform an ablation experiment to test the contribution of major architectural choices. The results are reported in Figure 3, and we provide methodological details in Appendix E in the supplemental materials.

We describe our hyperparameter selection procedure, which included an extensive sweep (see Table 5), in Appendix A.6 in the supplemental materials. Unfortunately, a full sensitivity analysis is prohibitively costly given the hyperparameter search space. We can, however, add to the paper a parallel coordinate plot or other data visualization for the configurations evaluated during our hyperparameter sweep.

TOTO incorporates variable modeling during pre-training. Since different datasets have different variables: a) How do you ensure that the number of variables in each batch of time-series data remains the same? b) Could varying numbers of variables across batches lead to difficulties in model pre-training?

Thanks for the question; it highlights an implementation detail which we mention briefly in Appendix A.2.1 but currently do not discuss in the main body. In pretraining, we use a fixed batch dimension of 32 variates and use document masking to pack multiple unrelated multivariate series in a single batch item. For series with more than 32 variates, we split these into multiple batch items.

At inference time, because the variate-wise attention layers are permutation invariant, we are able to scale up to as many variates as will fit into memory without needing any explicit extrapolation techniques. For example, in GIFT-Eval there are series with up to 321 variates, while BOOM contains series with up to 100 variates.

In Table 2, I believe it is necessary to evaluate smaller models to verify TOTO’s predictive performance. Simply comparing it with zero-shot Foundation Models is unfair because other Foundation Models were pre-trained on very limited observational time-series data, whereas TOTO used a much larger amount.

We do compare against deep-learning-based full-shot methods on GIFT-Eval, which is a multi-domain benchmark that is not primarily focused on observability data. We find that Toto’s zero-shot forecasts outperform full shot and small baseline models even when these competing models have been explicitly trained on the training splits of the datasets in GIFT-Eval (see Table 3).

To provide further empirical support, we have added evaluations on a subset of BOOM (what we call “BOOMLET” in the paper) using DLinear and DeepAR. These are two smaller full-shot models commonly used as baselines in the field. Interestingly, DLinear does outperform some other foundation models, including Chronos and Time-MoE. However, Toto still achieves the best overall performance by a wide margin on BOOMLET (we will update Table 2 accordingly):

We followed the same training setup proposed by the GIFT-Eval authors [10] for full-shot evaluation with hyperparameter tuning.

Table 3 lacks comparisons with other Foundation Models such as VisionTS and Time-MOE.

Table 3 is a reproduction of the GIFT-Eval leaderboard which includes our results. In the interest of saving space, we only printed the top several models in each category (zero-shot, full-shot, and baseline). With respect to VisionTS and Time-MoE:

• VisionTS is not competitive with the top models on GIFT-Eval, with MASE of 0.775, CRPS of 0.638, and Rank of 20.464 as of the time of our submission.

• Time-MoE has not submitted results to the GIFT-Eval leaderboard, and thus was not included in Table 3 (which as mentioned is a reproduction of the public leaderboard; we did not rerun models other than Toto).

We are happy to add a supplemental table with the complete GIFT-Eval leaderboard if the reviewer believes it to be necessary.

The main experiments on Foundation Models are all zero-shot evaluations. It would be beneficial to fine-tune TOTO and observe the effects under different levels of fine-tuning.

We did perform a fine-tuning experiment using the training portions of the LSF datasets, which we discuss in Section 5. The procedure and results are outlined in Appendix D.3, and summarized in Table 16. Our results show that finetuning results in an improvement Toto’s zero-shot performance, and achieves state-of-the-art performance over other full-shot methods. For convenience, we reproduce an excerpt of Table 16 showing the Toto's finetuned performance on the LSF datasets against the next-best full-shot model (TimeLLM):

Please do let us know if there are any remaining concerns that prevent you from increasing your score.

[1] Unified Training of Universal Time Series Forecasting Transformers. ICML 2024
[2] Mixture models and em. In Pattern Recognition and Machine Learning, pages 423–495. Springer, New York, 1 edition, 2006
[3] Adaptive Mixture of Student-T Distributions As a Flexible Candidate Distribution for Efficient Simulation. Journal of Statistical Software, Jan. 2009
[4] Robust mixture modelling using the t distribution. Statistics and Computing, 10(4):339–348, 2000.
[5] A general and adaptive robust loss function. CVPR 2019
[6] Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. ICLR 2023
[7] Msa transformer. ICML 2021
[8] Vivit: A video vision transformer. ICCV 2021
[9] A decoder-only foundation model for time-series forecasting. ICML 2024.
[10] Chronos: Learning the language of time series. TMLR 2024
[11] Gift-eval: A benchmark for general time series forecasting model evaluation. arXiv:2410.10393.

We thank the reviewer for their comments and provide clarifying discussion as well as new empirical support.

Novelty & Specialization to Observability

First, there were concerns raised around the novelty of Toto’s main components as well as their specific applicability to observability time series. We respectfully contend that our use of the Student-T Mixture Model (SMM) and the Hybrid NLL/Robust Loss for time series foundation models is novel; furthermore, the Causal Patch-based Instance Normalization algorithm is entirely novel. And while factorized attention has prior art, our specific Proportional Factorized Attention design offers meaningful computational efficiency gains tailored to high-cardinality observability data. We’ve organized our discussion per component to clarify what we feel makes them both novel and important for observability.

Student-T Mixture Model (SMM)

Novelty and comparison to Moirai: While other authors have used mixture models (e.g. Moirai), to our knowledge Toto is the first time series foundation model to use a Student-T Mixture Model head. This is a nontrivial improvement and generalization of the mixture model used in Moirai. As stated in their paper, Moirai uses a fixed mixture of four different distributions [1]. The Moirai authors suggest that these four individual components were chosen heuristically to match specific common cases within time series datasets.

By contrast, our model utilizes a mixture consisting of only Student-T distributions, where the number of components, $k$, is a hyperparameter that we optimize ($k=24$ in our selected model). This is fundamentally a more general and less heuristic approach. We derive our approach from first principles: mixtures of certain distributions (including Gaussians and Student-T’s) are universal approximators for any probability density function when using a sufficiently large number of mixture components [2, 3]. We choose the SMM (rather than GMM for example) based on its superior performance with heavy-tailed distributions [4].

In contrast to Moirai and other prior work, we also adopt a novel loss formulation which leads to significantly better convergence when training with our mixture model head. Alongside NLL, we jointly optimize the SMM with Robust Loss [4] to concentrate the mean towards accurate point predictions.

Significance for observability: As we show in Section 4.3, observability time series have challenging and diverse characteristics. The SMM’s expressive power is more important the further away we get from simple, well-behaved distributions. We use SMM rather than a GMM since we find that its heavy tails make it significantly more robust to outliers during training. Finally, the Robust Loss was chosen (rather than L1 or L2 loss) precisely because the presence of outliers in the training data makes these less-robust losses unstable.

Causal patch-based instance normalization

Novelty: To improve generalization across varying input scales, instance normalization is commonly applied prior to embedding time series data. However, computing normalization statistics from the entire series would leak information from future time steps; it violates the model’s autoregressive causality and results in poor performance from overfitting. To the best of our knowledge, this is a completely novel approach, and it yields highly non-trivial improvements, as demonstrated in our ablation.

Significance for observability: Preserving causality is especially critical in data which is highly nonstationary, like observability data (due to frequent regime changes and anomalies). The more the distribution shifts over time, the more future information is leaked by global statistics, which hurts generalization performance. Our ablation (Figure 3B) on observability data shows that causal patch-based normalization is the single most impactful architectural improvement of Toto.

Proportional factorized attention

Novelty: You correctly point out that factorized attention is not novel, and we do not make such a claim. While we agree that introducing an asymmetrical ratio of layers is a relatively simple idea, it nevertheless represents a meaningful improvement in computational efficiency of our model. Our asymmetric design is motivated by the observation that channel-independent models such as TimesFM [5] and Chronos [6] still achieve competitive performance on multivariate benchmarks and, therefore, we expect diminishing returns on FLOPs spent on variate-wise attention operations.

Significance for observability: Optimizing the variate-mixing of our model is particularly important for observability data, which typically has higher cardinality than other datasets. For example, our BOOM benchmark has a median number of variates of 60 (vs. 1 for GIFT-Eval and LSF).

We will revise the discussion in our Related Works and Model Architecture sections to make all of these points clearer.

Contribution of BOOM

Beyond the technical contributions described above, we would like to re-emphasize the significant contribution of BOOM, the first large-scale, comprehensive forecasting benchmark within the observability domain. In terms of absolute scale, it contains nearly twice as many points as GIFT-Eval; and as we show in Section 4.3, its distributional characteristics are significantly different from those of GIFT-Eval and LSF.

Other feedback / new results

The ablation experiments and analyses of individual components are overly simplistic. Merely removing them is insufficient to fully demonstrate the critical role each plays in the overall architecture.

We chose a “leave-one-out” ablation design as this allows us to isolate each novel architecture component, while saving significant compute resources vs. a full grid search. This methodology is commonly used in recent foundation model literature; for example, similar leave-one-out ablations are used by Moirai [1], Time-MoE [7], UniTS [8], PowerPM [9], and others.

Furthermore, we describe our hyperparameter selection procedure, which included an extensive sweep, in Appendix A.6 in the supplemental materials.

A comprehensive efficiency comparison with existing foundation models such as Moirai and Time-MoE is lacking.

Thank you for this feedback, we have conducted an empirical study of inference cost to help address this. We profile several models on synthetic data using a single A100-40GB, with a context length of 2,048 and a prediction length of 480, across 10, 50, 150, and 300 variates. We include comparisons against both multivariate (Moirai-Base) and univariate (Time-MoE-50M, Chronos-Bolt-Base, TimesFM-500M) models.

Toto consistently has the lowest FLOPs across all variate counts. Starting from 150 variates onward, Toto outperforms Moirai on all metrics; the gap grows significantly with more variates. Even against univariate competitors, where cross-variate interaction is ignored, Toto provides better performance, demonstrating its highly optimized architecture and efficient memory use. We note that Toto uses KV caching, while other models do not appear to implement this. For additional transparency, we also share Toto statistics with KV caching turned off and note that the general trend remains the same.

The results below show mean and std. dev. across 10 replications for each number of variates. (best, second-best)

What is the lookback window size used for the LSF benchmark?

We used a fixed lookback window size of 2,048. This is similar to Moirai, which uses lookback windows of 2,000 or 4,000 depending on the dataset.

Please do let us know if there are any remaining concerns that prevent you from increasing your score.

[1] Unified Training of Universal Time Series Forecasting Transformers. ICML 2024
[2] Mixture models and em. In Pattern Recognition and Machine Learning, pages 423–495. Springer, New York, 1 edition, 2006
[3] Adaptive Mixture of Student-T Distributions As a Flexible Candidate Distribution for Efficient Simulation. Journal of Statistical Software, Jan. 2009
[4] Robust mixture modelling using the t distribution. Statistics and Computing, 10(4):339–348, 2000.
[5] A general and adaptive robust loss function. CVPR 2019
[6] A decoder-only foundation model for time-series forecasting. ICML 2024
[7] Chronos: Learning the language of time series. TMLR 2024
[8] Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts. ICLR 2025
[9] UniTS: A Unified Multi-Task Time Series Model. NeurIPS 2024.
[10] PowerPM: Foundation Model for Power Systems. NeurIPS 2024.

Thank you for your thoughtful comments!

Potential overgeneralization in scope: While the paper presents strong contributions in the domain of observability data, the title and framing suggest broader applicability to general time series forecasting. Since both the model design and benchmark (BOOM) are heavily tailored to observability workloads, it may be somewhat overstated to position this work as a general-purpose time series foundation model without more extensive validation beyond this domain.

You correctly point out that the two main contributions of the paper, Toto and BOOM, are motivated by applications within the observability domain. And indeed, the architectural choices of Toto were designed with performance on observability data in mind.

However, we do find strong empirical evidence that Toto generalizes well to other domains. To demonstrate this, in addition to BOOM, we also provide evaluation results in Section 5 on GIFT-Eval $1$ and LSF $2$ . These are both widely accepted, general-purpose, multi-domain benchmarks that are not primarily focused on observability. They include many time series examples from non-observability domains, including finance, weather, energy, natural science, sales, and transportation. We feel that Toto’s state-of-the-art performance on these benchmarks clearly demonstrates Toto’s applicability outside of observability.

Although the authors state that there is no data leakage between the TOTO pretraining corpus and the BOOM benchmark due to environment isolation, both datasets are sourced from the same commercial observability platform. This raises concerns about distributional similarity. Since TOTO is pre-trained on large-scale observability data while competing foundation models are generally trained on more diverse or generic time series corpora, the performance advantage on BOOM may partially reflect domain alignment rather than purely architectural superiority.

While we did put significant effort into ensuring no data overlap between our training and test data by sourcing them from isolated environments, we acknowledge that there likely is some distributional similarity. That being said, as mentioned above, Toto also achieves state-of-the-art performance on GIFT-Eval and LSF, two general-purpose benchmarks containing time series primarily from domains outside of observability. As we show in Section 4.3 (“Statistical Characteristics”), these benchmarks are distributionally dissimilar from BOOM, suggesting that Toto’s strong performance is not solely due to in-domain evaluation.

Please do let us know if there are any remaining concerns that prevent you from increasing your score.

$1$ Gift-eval: A benchmark for general time series forecasting model evaluation. arXiv preprint arXiv:2410.10393, 2024

$2$ Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI, 2024

Thank you for the insightful comments. Below are the answers to your questions:

Please explain the data split in BOOM, given the heterogeneity of the data how the split was handled? And does the split happen at data point level or query level?

We describe the evaluation setting for BOOM, including details of the train/test split, in Appendix C.2 in the supplemental materials. To summarize, we adopt the standard practice within the forecasting literature of splitting each time series along the time dimension. Following the same setting used by GIFT-Eval $1$ , we designate the first 90% of each series as train and the last 10% as test. We do this split per query, meaning that each variate within a multivariate time series has the train/test split at the same timestamp.

Additional information regarding data preprocessing and quality controls for BOOM is provided in Appendix C.1.

How do you ensure the quality of the labels generated from the LLM and verified by humans is consistent? Please elaborate on the labeling approach, and if any inter-annotator agreement (for multiple humans) has been deployed.

LLM labeling was used to identify the subdomains (“Application”, “Network”, etc.) of the time series in the BOOM benchmark. This labeling does not impact the main results in Table 2, but rather only affects the taxonomy-level statistics provided in Figure 4 and the breakdowns in Tables 8 and 14 in the appendices. To clarify, the labeling followed a two-step protocol:

1. LLM Pre-labeling: system domains were initially labeled by a large language model
2. Human Review: The labels were then manually reviewed by multiple experts with domain knowledge. Each expert received a disjoint subset of the dataset to review. The experts consulted each other when they felt a particular example was ambiguous.

Please do let us know if there are any remaining concerns that prevent you from increasing your score.

$1$ Gift-eval: A benchmark for general time series forecasting model evaluation. arXiv preprint arXiv:2410.10393, 2024