GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation

Dear reviewer, thank you for the effort you put into reviewing our paper, and thank you for appreciating the scale of our collected data and the extensive amount of experiments we have conducted. We have carefully addressed each of your concerns and strengthened our presentation accordingly. Please find all responses below:

“W1: I believe that analyzing foundation models based on four time series characteristics—domain, frequency, prediction length, and the number of variates—combined with six time series features—trend, seasonality, entropy, Hurst, stability, and lumpiness—is not very meaningful, especially regarding the number of variates. I don't think it has any relation to these six features. Why not directly analyze the time series features for each test dataset, and then evaluate the performance of foundation models on various datasets to assess their strengths and weaknesses concerning these six features?”

Thank you for your comment. We believe analyzing models across these six time series characteristics is important mainly for two reasons: (1) Identifying a model’s dominant weaknesses provides valuable insights for improving both model architectures and datasets, e.g. some models designs may specialize in multivariate forecasting [1,4,5,6] or support for diverse frequency [1,2,3]; and (2) From a user perspective, understanding performance across specific characteristics (e.g., frequency or number of variates) helps in selecting the right model for their use case. For example, a user needing a weather forecasting model can prioritize forecasters that perform best on daily or weekly frequencies over second-level granularity.

That said, we agree with the reviewer that extending this analysis to include time series features for each test dataset is highly valuable. We have incorporated this into our paper, as detailed in the new section in Appendix F.1 and Table 16. This breakdown aggregates results based on time series features (e.g., trend, seasonality, entropy) rather than just dataset characteristics. Our findings show that deep learning models, particularly Transformer-based ones like PatchTST, excel in challenging scenarios with low temporal strength and high entropy, demonstrating strong generalist performance. In contrast, foundation models such as Moirai perform better in simpler, more predictable scenarios, consistent with Moirai-Large's strong results in less complex forecasting tasks. This also aligns with our aggregated results where Moirai ranked highly in majority of the forecasting scenarios yet PatchTST got better aggregated metric results as it can achieve reasonably well on all scenarios including challenging ones. We also note that these performance differences may partly reflect the supervised learning setups and hyperparameter tuning, which often favor deep learning models in more complex scenarios. We hope this extended analysis helps the community understand these nuances better without needing to re-analyze the data, especially as more models are added to our benchmark.

“W2: The paper only analyzes the features of the test data. It is necessary to include an analysis of the pretraining data as well. This would help better assess whether the foundation models perform well on these features due to the presence of these characteristics in the pretraining data itself, the generalization ability of the foundation models, or other reasons.”

Thank you for raising this important point. While we agree that analyzing the pretraining data would provide valuable insights into whether foundation models perform well due to characteristics in the pretraining data, their generalization ability, or other factors, conducting such an analysis is unfortunately beyond our current computational resources. For context, the analysis on the test data required approximately a week to complete using a compute server with 96 cores. Our pretraining data, however, is nearly 1,000 times larger than the test set, making such an analysis computationally infeasible with our available resources. We hope that by publicizing our pretraining dataset, other researchers with access to greater computational resources can contribute to this effort in the future. Meanwhile, our analysis of the test data provides meaningful insights into model performance across diverse characteristics.

Thanks to the author for their rebuttal. Although the author has addressed these questions, some issues remain unresolved. The main points are as follows:

• W1: The author only mentioned that analyzing the four time series characteristics is important but did not discuss the significance of combining time series characteristics—such as domain, prediction length, and the number of variates—with the six features—trend, seasonality, entropy, Hurst, stability, and lumpiness. I still believe the four characteristics are unrelated to the six features. Specifically, could the author explain the relationship between the number of variates and the six features?

  Additionally, the author's response referenced six related works, but three of them were not tested or compared.

• W3: Why can’t some small datasets be split using the standard splitting ratio? Is it because the input length is too long, resulting in a very small number of test samples after splitting, or is it due to the dataset itself being very small (with only a few hundred time stamps)?

• W4: According to the author’s response, the input lengths of foundation models and deep learning models are inconsistent. I believe the input length significantly impacts the model's performance, and this comparison may be unfair. Additionally, has the conclusion changed when using MSE and MAE as the metrics?

• W6: I still believe the author’s qualitative analysis lacks depth and still requires more meaningful conclusions. Additionally, the author mentioned that deep learning models excel in complex forecasting scenarios, while foundation models perform less well in such scenarios. This phenomenon contradicts the original intention of foundation models (due to pre-training on multi-source datasets, foundation models have strong generalization capabilities). Could you explain the reason behind this?

“Additionally, there is a lack of explicit connection with existing benchmarks to validate the reliability of the experiments conducted. For example, evaluation datasets in Tables 20, 21, and 22 cover some classical datasets like ETTh, Electricity, Solar, and M4. Comparing your results on shared datasets with existing studies could demonstrate the reliability of your experimental protocols. Moreover, some widely used datasets in the literature, such as Traffic, Wikipedia, and Exchange, appear to be excluded. Please clarify the reasons for their exclusion. I suggest maintaining existing classical benchmarks as they are while introducing new datasets and sampling frequencies that cover unique patterns. This approach allows for consistency with existing benchmarks, showcases the correct reproduction of existing models, demonstrates the necessity of new datasets and sampling frequencies, and reveals what can be discovered given this new benchmark.”

Excluding Traffic, Wikipedia, and Exchange

We appreciate the reviewer’s observation regarding the exclusion of Traffic, Wikipedia, and Exchange datasets. Traffic and Wikipedia are indeed included in the pre-training set to enrich the diversity and volume of training data. The Exchange dataset, representing financial time series, was excluded due to its unique characteristics. As noted in prior research [3], univariate financial time series often resemble a random walk, where the naive forecast is theoretically optimal. This behavior makes them unsuitable for general forecasting benchmarks, as it does not effectively challenge more complex forecasting models. While maintaining classical benchmarks could aid consistency, our focus is to demonstrate the added value of new datasets and diverse prediction lengths, revealing insights that existing benchmarks may overlook. We hope that by sharing our protocols and datasets, future studies can extend these findings and validate them further. We hope this clarifies our reasoning for dataset selection and the focus on benchmarks that can yield more meaningful model comparisons.

Connection with existing benchmarks

We appreciate the reviewer’s suggestion to draw explicit connections with existing benchmarks to validate our experiments. While we agree that comparing results on classical datasets e.g., ETTh, Electricity, Solar and M4 with prior studies would be valuable for demonstrating consistency and reproducibility, there are inherent challenges in doing so for the first three datasets as we use slightly different settings for these within our benchmarks:

1. In GIFT-Eval, we intentionally varied prediction lengths to enhance the diversity of forecasting scenarios. This decision was motivated by the limited diversity of prediction lengths in previous benchmarks, which often constrained models to specific forecasting horizons not representative of real-world applications. Our approach aims to create a more comprehensive evaluation by testing models across a wider range of forecasting conditions. However, this makes direct comparisons with existing studies, which use fixed or different prediction lengths, challenging.
2. Additionally, in line with similar motivations as the reviewer (incurring less costs to resource-limited research groups), we sample non-overlapping windows while ensuring at least 10% of each dataset is used in the test splits. This choice helps maintain an efficient evaluation process, as existing benchmarks often use a rolling window with a stride of 1, which is computationally expensive. These combined factors—varied prediction lengths and different test window setups—make raw dataset results harder to compare directly with existing studies. For M4 we kept the same settings as the original dataset, the only difference is that we filtered some very short datasets from our benchmark dataset thus yearly frequency is not identical to the original set. Below we share comparable results for all other frequencies from their original sources [1], [2]. (Table split into two responses due to space limitation)

Table Continued in the next repsonse..

[1] The M4 Competition: 100,000 time series and 61 forecasting methods, https://www.sciencedirect.com/science/article/pii/S0169207019301128

[2] Chronos: Learning the Language of Time Series, https://arxiv.org/abs/2403.07815

[3] Common Pitfalls and Better Practices in Forecast Evaluation for Data Scientists, Christoph Bergmeir. https://cbergmeir.com/papers/Bergmeir2023pitfalls.pdf

“Moreover, regarding three foundation models used in the experiments, only MOIRAI is re-trained on this new data split while TimesFM and Chronos are compared with their released model checkpoints. There is a risk of data leakage in these experiment comparisons. For example, TimesFM has leverage the Electricity dataset for pre-training while this dataset is also used for evaluation in this paper. This may explain why TimesFM performs extremely on "Electricity, short, H" (line 1285, Table 20), but this good performance may come from test data leakage. Therefore, to provide a fair and comprehensive comparison across TimeFM, Chronos, MOIRAI, and other representative time-series foundation models, re-training them following the new data split could be a necessary step.”

We acknowledge that re-training all foundation models on the new data split would provide a more comprehensive comparison. However, due to limited resources, we could only afford to re-train one model, Moirai, from scratch. It would be very limiting and shortsighted if we were to limit ourselves to only the datasets that current foundation models do not pretrain on. Instead we wanted our benchmark to be diverse across characteristics so that a broad evaluation can be achieved.

As noted in Section 4, other foundation models, such as TimesFM and Chronos, were compared using their released checkpoints, which may include data leakage into our test set (This is a standard issue that NLP benchmark papers acknowledge too [1]). However, we believe it is unrealistic to expect a single entity to re-train all foundation models, but by publicizing our pretraining datasets, we aim to facilitate collaborative scaling of this effort. The main reason for reporting retrained Moirai model results was to illustrate the impact of data leakage. However we understand that this may create an unfair standing compared to other models we incorporate in the experiments. Thus we update all tables to report results from the public Moirai model instead. We still report results of our findings over the retrained model in Appendix section F.3 to argue how leakage affects results.

[1] Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models https://arxiv.org/pdf/2206.04615

“The current analysis lacks depth across multiple experimental configurations. Presently, only aggregated comparisons are provided, without detailed analyses to derive new insights. For instance, why do some foundation models yield better zero-shot forecasting results on certain datasets than models specifically tuned for those datasets? Is it because pre-training datasets encompass more related patterns, pre-trained models have a larger model capacity, or classical models are not well-tuned, particularly regarding lookback length, modeling layers, or model designs? Merely presenting results without in-depth analysis can be detrimental to the research community, as others may expend significant efforts to re-analyze numerous results, which should be addressed within this benchmark.”

We appreciate the reviewer’s emphasis on deeper analysis and agree that providing more insights is valuable for the research community. Our primary motivation for this work is indeed to highlight the strengths and weaknesses of various model families to advance research in the time series domain. However, with 20 models evaluated across more than 20 datasets, explaining why specific model families perform differently on certain datasets is beyond the scope of this paper, as these are longstanding open research questions.

That said, we took the reviewer's suggestion to heart and expanded our analysis. In Appendix F.1 and Table 16, we provide a detailed breakdown that aggregates results based on time series features rather than just dataset characteristics, discussing the strengths and weaknesses of different model families. Briefly, our new findings indicate that deep learning models, particularly Transformer-based ones like PatchTST, tend to excel in challenging scenarios with low temporal strength and high entropy, showing strong generalist performance. Conversely, foundation models such as Moirai perform better in simpler, more predictable cases, aligning with our aggregated results where Moirai ranked highly in majority of the forecasting scenarios yet PatchTST got better aggregated metric results as it can achieve reasonably well on all scenarios including challenging ones. We also note that the performance differences likely reflect the supervised learning setups and hyperparameter tuning, which can give deep learning models an edge in more challenging forecasting tasks.

We hope this extended analysis will help the community better understand these nuances without having to re-analyze the data from scratch, especially with the growing number of models added into our benchmark.

Dear reviewer, thank you for the time you have taken to review our submission, and thanks for your kind words appreciating our research focus and the amount of experiments we conduct. We have carefully addressed each point below and strengthened our presentation accordingly. Please find all responses below:

The necessity of additional evaluation datasets

“MOIRAI [1] has already done an excellent job in collecting pre-training datasets and conducting comprehensive evaluations on widely recognized datasets typically used to evaluate time-series forecasting models in the literature. The datasets introduced in this paper heavily overlap with them while creating a new division for pre-training and downstream evaluation … For example, when incorporating a new dataset, does it cover distinctive patterns that rarely exist in existing benchmarks, thereby delivering unique insights on different pros and cons of model designs? The same question extends to the different sampling frequencies applied to each dataset. Is it necessary to include all sampling frequencies for every dataset? Does including the most prominent sampling frequency, which may be defined by different business scenarios, already provide sufficient and fair comparisons of different models?”

Thanks for your question. Constructing a benchmark requires the inclusion of datasets that collectively represent the complexity and diversity of real-world forecasting scenarios. While explaining the specific contribution of each dataset in isolation might not be feasible, GIFT-Eval was curated with the principle of achieving diversity and ensuring adequate coverage of key characteristics such as domain, frequency, and prediction length, including multivariate scenarios. This selection was based on publicly available datasets with careful balancing to maintain diversity in the test set while also reserving key datasets for effective pretraining.

The reviewer rightfully asked whether sampling various frequencies of some datasets is a necessary step. Incorporating multiple sampling frequencies tests a model’s adaptability and robustness across different business scenarios, which would be missed if only the most common frequency were used. Even through our results one can observe that a model showing promising results for a certain frequency may give relatively poor results on the same frequency for some datasets. We agree with the reviewer that Moirai’s pretraining dataset is extensive, yet its evaluation data lacks the diverse scope needed for fully assessing model versatility. We show why we think so in the next section.

Comparison to Moirai

Moirai Zero-shot datasets: Prob. Forecasting

Long Term Forecasting

Moirai's original zero-shot evaluation employed a limited set of datasets, primarily focusing on specific domains and frequencies. Below we provide two example comparison with Moirai’s evaluation to argue why Gift-Eval is more comprehensive:

a. Long term Forecasting: Original Moirai results demonstrated its strength in long-term forecasting (outperforming baselines in 5 out of 6 datasets), these findings align with our observations for those specific datasets. However, GIFT-Eval introduces 21 dataset configurations with long term forecasting across diverse domains and frequencies. Deep learning models like PatchTST and iTransformer, which were also included in Moirai's comparisons, outperform Moirai in long-term forecasts within our benchmark.

b. Frequency We can look at another example in favor of Moirai from the frequency perspective. Moirai's original evaluation included only the Walmart dataset at a weekly frequency, where it underperformed compared to deep learning models. In contrast, GIFT-Eval includes 8 weekly datasets from 4 different domains with both univariate and multivariate representations. In this broader context, Moirai performs significantly better, surpassing all deep learning baselines and securing second place among foundation models.

These examples underscore why diverse benchmarks like GIFT-Eval are essential for accurately evaluating the general capabilities of universal models. They uncover performance dynamics that narrower evaluations might miss, providing a broader picture of model strengths and weaknesses.

Dear reviewer, thank you for appreciating the scale of our collected data and recognizing the problem our benchmark addresses for the time series forecasting community. We have carefully addressed each of your concerns and strengthened our presentation accordingly. Please find all responses below:

“My major concern of the work: MAPE is well recognized as a bad point forecasting metric on its own, due to e.g. its favoring underprediction, sensitivity near ground truth 0 (this is recognized by the authors too), (see e.g., [1] and many community posts online). It would be helpful to switch to or also report other more robust metrics, e.g. MASE.”

“In case normalized metrics are used, consider reporting geometric means [2].”

Thank you for your suggestion. Other reviewers have also brought up similar concerns. Following all reviewer suggestions we omit MAPE as an evaluation metric from our paper. Each reviewer had a different suggestion for metrics to add. Here is how we addressed all of their concerns:

The main paper tables (Table 2 through 7) in the paper are updated to show MASE rather than MAPE results and we use geometric mean to aggregate results instead of arithmetic mean. We update the appendix tables (Tables 17 through 21) which report results for all models with all metrics suggested by reviewers for the sake of completeness. Specifically we report results with sMAPE, MASE, ND, MSE, MAE, CRPS and finally Rank metrics.

We hope this addresses the concerns around the evaluation metrics.

“Given the 6 properties of component datasets, it would be helpful to see the benchmark results sliced on these properties as well.”

Thank you, this is a great suggestion. We report these results in a new section in Appendix F.1 and Table 16, where we provide a detailed breakdown that aggregates results based on time series features rather than just dataset characteristics, discussing the strengths and weaknesses of different model families. Briefly, our new findings indicate that deep learning models, particularly Transformer-based ones like PatchTST, tend to excel in challenging scenarios with low temporal strength and high entropy, showing strong generalist performance. Conversely, foundation models such as Moirai perform better in simpler, more predictable cases. This also aligns with our aggregated results where Moirai ranked highly in majority of the forecasting scenarios yet PatchTST got better aggregated metric results as it can achieve reasonably well on all scenarios including challenging ones. We also note that the performance differences likely reflect the supervised learning setups and hyperparameter tuning, which can give deep learning models an edge in difficult predictions.

We hope this extended analysis will help the community better understand these nuances without having to re-analyze the data from scratch, especially with the growing number of models added into our benchmark.

Dear reviewer, thank you for taking the time to review our submission and for recognizing our benchmark as a promising means for evaluating foundation models. We greatly appreciate your positive feedback on the scale of our collected data and the extensive experiments we conducted. We have carefully addressed each of your concerns and strengthened our presentation accordingly. Please find all responses below:

“This paper emphasizes the inclusion of a non-leaking pretraining dataset, but its value and usage are not clear. Is this dataset used to re-train all foundation models instead of using their public checkpoints? Is it necessarily better than the original pretraining datasets of each foundation model? Since the application on downstream data is the main goal of foundation models, do we really need to keep consistency in pretraining to evaluate these models as discussed in the Introduction?”

Thank you for the question. The purpose of providing pretraining data is not to claim it is superior to the original datasets used by each foundation model, nor to prescribe its usage for new pretraining efforts. Rather, it ensures researchers have access to a non-leaking pretraining dataset with our evaluation set. In our paper, we demonstrate the efficacy of this dataset by re-training Moirai variants and comparing its public version with the retrained version in Appendix F.3. While re-training all foundation models on the new data split would provide a more comprehensive comparison, resource constraints allowed us to re-train only one model from scratch.

For fairness, all tables except F.3. use public versions of foundation models. As noted in Section 4, many of these models may involve some level of data leakage into our test set. However, limiting the benchmark to datasets untouched by existing pretraining efforts would severely restrict its diversity and utility. Our focus is on creating a diverse benchmark for broad evaluation, even if this involves datasets overlapping with some pretraining sets. This is an issue that has also been acknowledged by NLP benchmark papers and addressed similarly [2].

Finally, while it is unrealistic for a single entity to re-train all foundation models, publicizing our pretraining datasets facilitates collaborative scaling of this effort. This approach was also implemented in other NLP benchmark papers [1].

[1] XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation, https://aclanthology.org/2020.emnlp-main.484/

[2] Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, https://arxiv.org/pdf/2206.04615

“Section 3.1.1 mentions covariates in time series forecasting, but it is unclear how are the covariates considered in this benchmark. Can all baselines in the benchmark perform forecasting with covariates?”

Thank you for this insightful question. We believe that support for covariates is an advantage that models capable of utilizing them should leverage. Accordingly, in our experiments, we include covariates for models that support them, while others rely solely on the input variate. We noticed that our paper is missing number of covariates information for test datasets, we updated Table 14 with the relevant information in our paper.

Thank you for providing the rebuttal. Some of my concerns have been addressed. Here are some responses:

W1: Now I understand the role of the non-leaking pretraining dataset. It seems that currently data leakage still exists in the evaluated models. How would this influence the fair comparison between foundation models with diverse data leakage?

W2: Could you please point out which parts in Table 14 are the covariates and provide some examples to show what these covariates exactly are? Do all the evaluated datasets have covariates?

W4: The mentioned analysis is based on the dimension of six different features, which is a little confusing considering the other dimension of four different characteristics. Why should we use these two different dimensions for analysis? Is one of them more intrinsic for time series data? Additionally, the number of foundation models considered in this paper is limited considering the emergence of such models.

W6: Why do Figures 2(a) and (c) use different data for visualization? It is unclear why we can ensure that these samples can identify prominent issues.