ProbTS: A Unified Toolkit to Probe Deep Time-series Forecasting

Thank you for the response. My thoughts and score currently remains the same.

1. Could you explain concretely what you mean by "ProbTS as a unique research-oriented tool that unifies divergent yet equally important research in deep time-series forecasting, reveals overlooked challenges, and opens up new avenues for future research."? What are the features or toolkit design that allows ProbTS to be so unique?
2. I disagree with the characterization that "we find that GluonTS is highly wrapped and leans toward probabilistic forecasting, making it difficult to incorporate different data preprocessing, normalization, and evaluation implementations used by existing studies in neural architecture design". GluonTS has an extensive data pre-processing pipeline (https://github.com/awslabs/gluonts/tree/dev/src/gluonts/transform), supports many normalization techniques (https://github.com/awslabs/gluonts/blob/dev/src/gluonts/torch/scaler.py), and has a full suite of evaluation metrics (https://github.com/awslabs/gluonts/tree/dev/src/gluonts/ev). It is also easily extendible, allowing to add new transforms, models, and evaluation metrics.
3. It is difficult to evaluate the benefits of ProbTS without looking at the code or any design documents about the library.
4. GluonTS already implements 2 of 4 of the "advanced neural architectures" mentioned, PatchTST and DLinear.
5. pytorch-transformer-ts uses GluonTS to implement many more "advanced neural architectures", and it does probabilistic forecasting.
6. The paper is very limited by not analyzing the interplay between architecture and probabilistic head. How can we disambiguate the effects of architecture and probabilistic head? Does performance stem from different architecture or head? Yes, it is not straightforward, but that would constitute a significant contribution rather than simply benchmarking different methods which arguably are not fair comparisons with each other.

Thank you for your thorough review of our paper. We apologize for any misunderstanding and lack of clarity regarding the strengths of our work. We hope the following clarifications and explanations can help address your concerns.

Response to Weakness 1

We would like to clarify that the primary contribution of this paper is to unify two key aspects in deep time-series forecasting: customized neural architecture designs and advanced probabilistic estimations tailored for time series. This work is set apart from existing studies in that we strive to unify these two aspects and expose underexplored yet crucial topics, accompanied by our findings and insights, to the research community. Specifically, this paper discloses and analyzes several unanswered questions:

• How do different data characteristics and forecasting horizons influence methodological designs in previous studies? What are the implications when evaluating them in underexplored scenarios?
• What are the strengths and weaknesses of either focusing on neural architecture designs or emphasizing advanced probabilistic estimations? How will existing approaches perform on different evaluation metrics, when forecasting distributional or point-level forecasts?
• Why does the development of these two research branches lead to different preferences on decoding schemes, either autoregressive or non-autoregressive?

We've briefly summarized our insights towards these questions in the introduction of our paper and included more detailed results and analyses in Section 4. What we want to clarify and emphasize most is that ProbTS is not a traditional tool aiming to consolidate all library attributes, such as metrics, datasets, data transformations, and data loading efficiency, into one place. Instead, we envision ProbTS as a unique research-oriented tool that unifies divergent yet equally important research in deep time-series forecasting, reveals overlooked challenges, and opens up new avenues for future research.

In response to your constructive suggestions regarding the wording of the "Model" column in Table 1 and the additional discussions between this work and existing tools, we have made appropriate adjustments. For instance, we have changed the "Model" column to "Studies", indicating that each entry in this column represents an individual paper studying specific problems. We have also compared ProbTS with existing tools from various perspectives, as shown in the following table.

Indeed, GluonTS is an excellent tool that comes closest to ours. However, when we attempt to unify two different research branches, we find that GluonTS is highly wrapped and leans toward probabilistic forecasting, making it difficult to incorporate different data preprocessing, normalization, and evaluation implementations used by existing studies in neural architecture designs. As a result, we decided to develop ProbTS from scratch as a research-oriented tool specializing in unifying neural architecture designs and probabilistic estimations in deep time-series forecasting.

[1] Alexandrov, Alexander, et al. "Gluonts: Probabilistic time series models in python." arXiv preprint arXiv:1906.05264 (2019).

[2] https://github.com/thuml/Time-Series-Library

[3] https://github.com/kashif/pytorch-transformer-ts

[4] https://github.com/unit8co/darts

[5] https://github.com/salesforce/Merlion

[6] https://github.com/facebookresearch/Kats

Response to weakness 2 & 3

Thank you for your comments. We appreciate your observations and would like to emphasize that our evaluation does indeed provide a robust comparison of existing methods across two distinct research branches.

Firstly, for our benchmarking, we have included the most representative and top-performing baselines from each branch. This includes highly competitive neural architectures such as PatchTST, TimesNet, D-linear, N-HiTS, as well as recent advanced probabilistic forecasting approaches like CSDI, TimeGrad, and Trans MAF. We are certainly open to incorporating additional methods into our comparisons. If you could suggest any that surpass the performance or possess unique advantages over these existing baselines, we would be more than willing to consider them.

Secondly, we've created some ablation variants by removing the probabilistic head from GRU NVP and Trans MAF, which are represented by the GRU and Transformer models in Table 4. Concerning your proposal of stacking probabilistic heads on customized architectures, such as Autoformer and PatchTST, it's crucial to note that this isn't a simple undertaking. It either necessitates transforming these non-autoregressive architectures to the autoregressive scheme (as seen in GRU MAF, GRU NVP, Trans MAF, TimeGrad) or integrating probabilistic modules in a non-autoregressive manner. For example, CSDI combines architecture designs with non-autoregressive probabilistic estimations. However, CSDI struggles in long-term forecasting due to its increasing inefficiency as the forecasting horizon expands, and its inability to handle long trending or strong seasonality effectively.

In summary, the task of combining customized architectures with advanced probabilistic estimations presents many unresolved challenges and is not as simple as adding a probabilistic head over an existing architecture. After disclosing and analyzing these remaining problems in this paper, we believe there are many opportunities ahead to explore how we can better combine customized architectures with advanced probabilistic estimations while addressing the limitations of existing probabilistic forecasting methods. In this regard, ProbTS aims to serve as a foundational tool to facilitate future research.

It seems from the latest response that the positioning of ProbTS is now as an empirical study, rather than a toolkit, so I shall now give my thoughts from the perspective of an empirical study (and therein lies my original critique -- this paper has split its focus between a library/toolkit, benchmark, and empirical study, failing to hit the mark on any).

1. Findings regarding datasets are not well-supported.

   a. Exchange-S/L, Electricity-S/L, and possibly Traffic-S/L are exactly the same datasets, Solar has strong seasonality -- it is not clear how the data supports the conclusion that long term forecasting datasets have more trend/seasonality than short term forecasting datasets.

   b. How does the JS div score lead to the conclusion that the datasets have more complex data distributions? Does this mean that all non-gaussian data distributions are more complex? What about a Student-T distribution head or Negative-binomial distribution head? These are simple modelling choices that can handle non-Gaussian data.

2. Findings regarding models do not constitute significant contributions.

   a. "Probabilistic methods excel in modeling complex data distributions" -- probabilistic methods obviously are better than point forecasting methods at modelling distributions.

   b. "Customized network architecture performs remarkably on long-term forecasting." -- similarly, these "customized network architectures" are methods proposed specifically for the LSTF setting, and naturally performs better on the LSTF setting.

   c. "Probabilistic methods may produce poor point forecasts even with a superior CRPS score" -- again, these findings are based on very shaky experimental design. How do we know probabilistic methods are worse than point forecasting methods on the NMAE metric due to it being a probabilistic method, rather than the architecture design? It has superior CRPS score due to the nature of the metric favouring probabilistic forecasts, but if you gave the more advanced architecture design a probabilistic head, would it have gotten a superior CRPS score? If you want to make such claims, the experiments have to be carefully designed, performing ablations on the different components. We cannot draw such conclusions from the benchmark presented.

   d. "Customized network architecture on short-term forecasting remains underexplored" -- If you consider that the "customized network architectures" have mostly been proposed for the LSTF task, then this statement is straightforward and does not require experiments to arrive to

Response to Question 1

As outlined in Appendix C.1, we adopted an exhaustive grid search approach for model tuning. We acknowledge your valuable suggestion to employ standard hyperparameter tuning packages. In future iterations, we will investigate incorporating such packages to improve the efficiency of parameter tuning, particularly when introducing additional baselines.

Response to Question 2

To ensure a fair comparison, we set the hyperparameters via thorough standard tuning procedures. For those models and datasets where hyperparameters were previously provided, we reused the reported patch size that had been demonstrated to deliver optimal results, a finding that our reproduction also validates. When it came to short-term forecasting and patch size settings were not available, we conducted a search for the ideal patch sizes within a range of 4 to 8. This was done considering the shorter lookback window and forecasting horizon.

Response to Question 3

In most instances, where detailed reproducible specifications were provided (for both datasets and models), we successfully reproduced results by directly applying or slightly adjusting the reported hyperparameters. For those scenarios without clear guidance, such as evaluating probabilistic methods in long-term forecasting scenarios, we conducted a unified grid search over some critical hyperparameters on the validation set to determine the final experimental setups.

However, it's important to approach comparisons of our results with those in the original papers cautiously due to subtle differences in metric calculations. Some methods denormalized their experimental results before evaluation [1,2,3], while others calculated the metric scores directly without denormalization [4,5], creating a challenge for direct comparison. To address this, our toolkit preserves both denormalized and non-denormalized results, facilitating a unified evaluation and comparison. In our main paper, we report the evaluation metrics calculated at the original value scale (after denormalization).

We made every effort to ensure a fair evaluation of all baselines, and we plan to open-source all the code and configuration files. This commitment to fairness underpins the insights we derived, which form the primary contribution of our paper. We appreciate your understanding of these complexities and welcome further suggestions to enhance the transparency and robustness of our comparisons.

[1] Rasul, Kashif, et al. (2021). Multivariate probabilistic time series forecasting via conditioned normalizing flows. Proceedings of the ICLR.

[2] Rasul, Kashif, et al. (2021). Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. Proceedings of the ICML.

[3] Tashiro, Yusuke, et al. (2021). CSDI: Conditional score-based diffusion models for probabilistic time series imputation. Proceedings of the NeurIPS.

[4] Wu, Haixu, et al. (2022). Timesnet: Temporal 2d-variation modeling for general time series analysis. Proceedings of the ICLR.

[5] Nie, Yuqi, et al. (2022). A time series is worth 64 words: Long-term forecasting with transformers. Proceedings of the ICLR.

Response to Weaknesses 2

Thank you for bringing up the issue of modeling cross-channel interactions in time series forecasting. We have given this aspect careful thought, but as of now, we haven't discerned a clear pattern relating the modeling of cross-channel interactions to overall model performance. We have prepared a summary table illustrating how models from each branch handle the multivariate aspect, as detailed below.

Table. Summary of how existing models handle multivariate time series.

A notable trend is the prevalent use of a channel-mixing approach in most studies. However, findings are diverse, with models like DLinear [1] and PatchTST [2] suggesting that processing channels independently can yield superior results, while others like CSDI [3] indicate that explicit modeling of cross-channel interactions offers significant advantages. This diversity points to the ongoing exploration of the impact of cross-channel interactions on forecasting performance.

Additionally, we have consistently noted that as dimensionality escalates, the modeling of cross-channel interactions necessitates greater memory capacity, especially for methods based on probabilistic estimation.

We sincerely value your insightful feedback, and we plan to incorporate a discussion on this topic in our revised manuscript. We hope this will stimulate more in-depth investigations into this significant aspect of time series forecasting.

[1] Zeng, Ailing, et al. (2023). Are transformers effective for time series forecasting?." Proceedings of the AAAI. Vol. 37. No. 9.

[2] Nie, Yuqi, et al. (2023). A time series is worth 64 words: Long-term forecasting with transformers. Proceedings of the ICLR.

[3] Tashiro, Yusuke, et al. (2021). Csdi: Conditional score-based diffusion models for probabilistic time series imputation." Proceedings of the NeurIPS.

We are deeply grateful for your acknowledgment of the unique contributions made by this work. Your recognition of our analyses as both interesting and insightful is highly appreciated. In what follows, we will address your remaining concerns and questions specifically.

Response to Weaknesses 1

We appreciate your emphasis on the significance of comparing methods across datasets of varying sizes. Your viewpoint on this matter aligns with similar insights raised in Weakness 2 by Reviewer eUu7 and Question 2 by Reviewer AnhU. For further details, please refer to our responses to their queries. In the following, we present a comprehensive response to your question.

First, we would like to clarify that the datasets currently included in ProbTS already cover a wide range of sizes. Below is a table summarizing the statistics of these datasets. As demonstrated, these datasets span a wide range of scales, from smaller sets containing 792 timesteps to larger ones with 69,680 timesteps, and in their dimensionality, ranging from low (7) to high (2000). We believe this wide array of datasets adequately substantiates our findings.

Table 1. Statistics of datasets.

Furthermore, we have undertaken a quantitative analysis to discern the relationship between the data scale and the superiority of probabilistic methods. Our analysis revealed that dataset size does not play a pivotal role in influencing the design paradigm of the two branches under consideration. In our preliminary analysis, we examined the impact of data volume on performance and did not identify any clear correlation patterns, as demonstrated in the tables below.

Table. The correlation coefficient between the data volume and the relative performance improvement of CRPS/NMAE compared to the baseline model (GRU).

Thank you for your comprehensive review and constructive feedback. We also particularly appreciate your recognition of our work. We hope that the following responses will address your concerns and contribute to enhancing the completeness and rigor of this paper.

Response to Weaknesses 1

We greatly appreciate your suggestion and welcome any specific recommendations you may have to further highlight these insights. In the submitted manuscript, we briefly outlined these insights in the Introduction section and provided detailed discussions in Section 4. The key insights of each paragraph have been accentuated by bolding them and placing them at the beginning of each section. In the upcoming revision, we plan to improve our writing to ensure that the core insights of each paragraph are conveyed more effectively to the reader.

Response to Weaknesses 2

We appreciate your suggestion about the need for a more extensive discussion on the critical characteristics of time-series forecasting, such as dimensionality, data length, and the volume of training data. We did examine these factors in our preliminary studies, but they did not profoundly influence the design paradigm of the two branches under consideration.

We did not observe a significant correlation between these factors and model performance, as indicated in the tables below. As these findings did not provide substantial insights, we chose not to include them in the paper. Nevertheless, we believe the ProbTS Toolkit is well-equipped to facilitate more in-depth analyses of these characteristics in future research if needed.

Table 1. The correlation coefficient between the data volume and the relative performance improvement of CRPS/NMAE compared to the baseline model (GRU).

Response to Weaknesses 3

We appreciate your constructive feedback. We concur that controlled experiments and the use of synthetic datasets could provide more robust empirical evidence. Consequently, in response to your suggestion, we have generated synthetic datasets including a baseline dataset, one with pronounced trends, another with strong seasonality, and a dataset displaying complex data distribution. Each series within these datasets was created by combining trend, seasonality, noise, and anomaly components with controlled characteristics. The details are provided in the table below.

Table 2. Quantitative assessment of intrinsic characteristics for synthetic datasets.

Subsequent experiments conducted on these synthetic datasets, utilizing representative models, validated our empirical findings from other datasets examined with ProbTS. Noteworthy observations include the declining performance of autoregressive decoding models such as TimeGrad in the presence of increasing trends, an enhancement in performance for models employing autoregressive decoding with intensifying seasonality, and the competitive performance exhibited by probabilistic methods like CSDI when dealing with more complex data distributions.

Table 3. Results on synthetic datasets. The look-back window and forecasting horizon are 30.

These observations further reinforce our findings, and we plan to incorporate these synthetic experiments and analyses in future revisions of our work. In addition, we aim to include the code for generating synthetic datasets within the ProbTS toolkit to encourage and facilitate further exploration of this area by the wider research community. We greatly appreciate your valuable feedback!

Response to Weaknesses 4

Thank you for your suggestion. We will take care to ensure that all abbreviations, including CRPS, are clearly defined prior to their first use in the revised version of the paper.

Response to Question 3

Your question is indeed pertinent. As specified in Appendix C.1, we carried out an extensive grid search for models, tuning hyperparameters individually for each method. Given the large number of models and the variance in hyperparameters across different datasets and forecasting scenarios, we chose not to include a detailed account of parameter configurations in the main text of the paper. However, we have made the specific hyperparameter configurations available in an open-source format along with the code.

In response to your valuable suggestion, we will include the key hyperparameter settings in the upcoming revision to offer readers a more accessible reference point. Thank you for raising this important issue.

Response to Question 4

We appreciate your thoughtful question. The categorization of datasets into short-term or long-term forecasting is primarily influenced by practical necessities in various domains, rather than our personal determination of which dataset is best suited for a particular forecast duration. For instance, certain datasets like Wikipedia are commonly utilized for short-term forecasting. Others, such as Weather-L and ETTm1-L / ETTm2-L, are typically employed for long-term forecasting. Some datasets, like Electricity and Traffic, are applicable to both short-term and long-term forecasting.

In this paper, our aim is not just to distinguish between short-term and long-term forecasting or varying domains. Instead, we strive to identify inherent dataset characteristics that may guide different methodological designs. The key characteristics we have identified include the strength of trending, the strength of seasonality, and the complexity of data distribution. These traits are a collective result of the length of forecasting horizons and the specific data domains.

Our exploration into the interplay between data characteristics and model designs prompts us to envision future research that would conduct more comprehensive evaluations. These evaluations would span across different domains and forecasting horizons (short versus long), and would utilize a variety of evaluation metrics, both at the distributional and point level. We believe that ProbTS is well-equipped to facilitate this kind of comprehensive analysis. We value your feedback and will strive to clarify our presentation further in the next revision.

Response to Question 5

Thank you for your thoughtful question. We appreciate the opportunity to elucidate the synergistic relationship between model and data aspects in our research.

Firstly, our work does indeed shed light on the interplay between these two aspects. We have found that specific data characteristics favor certain methodological designs, leading to a convergence in different studies, a phenomenon we have outlined in our paper.

Secondly, we have evaluated models in data scenarios that have been scarcely studied in the past. This approach has allowed us to uncover overlooked challenges and unresolved problems in the field, thus contributing to a more holistic understanding of the topic.

For example, our work includes the following findings.

• Probabilistic methods demonstrate proficiency in modeling complex data distributions, even though they may produce poor point forecasts even with a superior CRPS score (Section 4.2).
• Customized network architecture shows remarkable performance in long-term forecasting, yet it remains an underexplored area in short-term forecasting scenarios (Sections 4.3).
• Autoregressive models excel in handling strong seasonality but face challenges with pronounced trends, while both autoregressive and non-autoregressive decoding schemes perform comparably well in short-term forecasting (Section 4.4).

Lastly, the intricate interplay between data and model is a persistent theme in machine learning research. We have developed ProbTS in the hope that it will facilitate a deeper exploration of this dynamic in the time-series domain. ProbTS achieves this by providing comprehensive evaluation scenarios and metrics, and by unifying essential modeling components.

We hope this clarifies the intent and findings of our work. Thank you for prompting this valuable discussion.

Response to Question 1

We are grateful for your thoughtful suggestion to include traditional tree-based methods, such as XGBoost, in our comparison. Following the implementation provided by [1], we found that XGBoost indeed exhibits competitive performance, even surpassing neural network (NN) models in certain scenarios.

However, for datasets with more complex data distributions such as Solar and Electricity, advanced probabilistic estimation methods demonstrated a significant advantage over both traditional learning methods and point estimation methods. This underscores the versatility and strength of these advanced probabilistic methods in handling complex forecasting scenarios.

Despite the competitive performance of XGBoost, one of its limitations is its reliance on handcrafted features, which contradicts the recent trend toward automatic representation learning inherent in NN models. This is one of the key reasons why, in line with many recent studies, we've chosen to focus on deep time-series forecasting.

Furthermore, it's important to note that both tree-based methods, such as XGBoost, and NN models necessitate extensive hyper-parameter tuning. Despite our rigorous hyper-parameter tuning efforts, we did not achieve satisfactory performance with XGBoost on the Wikipedia dataset. This outcome may suggest that additional feature engineering is required for this specific dataset. However, considering the time constraints and the substantial efforts involved in preparing responses, we have decided to exclude this dataset from the subsequent table.

Your insightful suggestion has undoubtedly enriched our study. By demonstrating the strengths of both XGBoost and NN models, we've provided a more comprehensive view of the current landscape of time-series forecasting. We believe this expanded perspective will enhance the overall depth and value of our paper. We greatly appreciate your contribution to this improvement.

[1] Elsayed, S., et al. (2021). Do we really need deep learning models for time series forecasting?. arXiv preprint arXiv:2101.02118.

Response to Question 2

We appreciate your insightful question. ProbTS has been designed and tested on a diverse range of datasets, as illustrated in the table below. These datasets vary significantly in scale, ranging from small (792) to large (69,680), and in dimensionality, from low (7) to high (2000). We believe this broad spectrum of datasets adequately validates our findings.

Table 1. Statistics of datasets.

In addition, we specifically investigated the impact of dataset size on model performance. Our analysis, depicted in Table 3, did not reveal a clear correlation between dataset size and model performance. Consequently, this aspect was not emphasized in our paper.

Table 2. The correlation coefficient between the data volume and the relative performance improvement of CRPS/NMAE compared to the baseline model (GRU).

Thank you for your insightful comments and for recognizing some unique contributions of our work. We would like to further clarify our motivation and the core position of this work to enhance your understanding. We regret any misunderstanding and would like to assure you that this work is not just about combining long-term and short-term forecasting.

Firstly, the primary motivation of this work is to unify two equally important yet independently developed research branches in deep time-series forecasting:

• One branch focuses on customized neural architecture designs but only provides point-level forecasts.
• The other branch specializes in probabilistic estimations for distributional forecasts but rarely includes architecture designs.

Secondly, during this unification research, we identified several critical aspects that need to be aligned, including:

• The role of different data characteristics (trend, seasonality, complexity of data distribution) and forecasting setups (long-term vs short-term) in influencing the methodological designs and choices.
• The existence of several unexplored scenarios and overlooked challenges in the literature due to different methodological designs in rarely visited data scenarios. The significant methodological aspects we've summarized include:
  • Probabilistic vs non-probabilistic forecasting paradigms and evaluation metrics.
  • Customized vs general neural architecture designs.
  • Autoregressive vs non-autoregressive decoding schemes.

Thirdly, after conducting experimental comparisons and analyses of these methodological designs across diverse data scenarios, we present our unique findings and insights on the strengths and weaknesses of different methods. We believe the challenges we uncovered and the new opportunities we identified will pave the way for future research aimed at better combining the benefits of neural architecture designs and probabilistic estimation abilities.

We hope the above clarification of our paper's logic can help to reinforce your understanding of the unique contributions of our work. With this context, we will proceed to address your specific concerns in more detail.

Response to Weaknesses

We appreciate your reference to these important studies on time-series benchmarking. We plan to incorporate a discussion on these works in our revised paper. However, it's crucial to highlight the fundamental differences between our work, ProbTS, and these studies.

In brief, [1] provides a broad range of datasets and benchmarks a multitude of forecasting methods, [2] primarily compares gradient boosted decision trees with deep learning methods, and [3] aims to categorize existing methods in different aspects, appearing more like a literature review. Specifically, neither [1] nor [2] consider probabilistic forecasting, while [3] focuses on fine-grained classifications of forecasting methods beyond the simple "machine learning" or "statistical" dichotomy.

It's noteworthy that these studies were published at least two years ago. Given the rapid development in this field, our work is unique in its recognition of the need to unify the research in customized neural architectures and advanced probabilistic estimation abilities, both of which are vital components in deep time-series forecasting but have evolved independently in recent literature. In this context, our work reveals several unique insights and findings, such as:

• The distinction between long-term and short-term forecasting, along with different data characteristics.
• Probabilistic vs non-probabilistic forecasting paradigms and evaluation metrics.
• Customized vs general neural architecture designs.
• Autoregressive vs non-autoregressive decoding schemes.

In summary, our work is not merely another benchmarking study nor a comprehensive review covering all aspects of time-series forecasting. We view our work as a critical step in bridging the prominent yet divergent branches in contemporary research on time-series forecasting.

[1] Godahewa, Rakshitha, et al. (2021). Monash time series forecasting archive. arXiv preprint arXiv:2105.06643.

[2] Elsayed, S., et al. (2021). Do we really need deep learning models for time series forecasting?. arXiv preprint arXiv:2101.02118.

[3] Januschowski, Tim, et al. (2020). Criteria for classifying forecasting methods. International Journal of Forecasting, 36(1), 167-177.

Response to Weaknesses 3

Thank you for your insightful observation regarding the scale of the datasets used. We concur with your assertion that the inclusion of large-scale datasets is crucial for future research development. As per your recommendation, we plan to progressively incorporate significant, distinctive, and large-scale datasets into ProbTS, starting with the FRED dataset collection.

In the meantime, we would like to underscore that the diversity of data is another crucial aspect that warrants thorough investigation, which is the primary focus of this paper. Evaluating learning ability across different data scales, variate numbers, and domains with a wide scope of data characteristics is of great importance. Therefore, this paper is dedicated to evaluating diverse data characteristics and conducting comprehensive analyses to shed light on overlooked challenges and unresolved issues.

The datasets included in this paper are listed below. As can be seen, ProbTS already accommodates a wide range of data scales, both in terms of the number of timesteps (ranging from 792 to 69,680) and the number of variates (ranging from 7 to 2,000). Please note that most existing studies only employ a subset of these datasets to evaluate the effectiveness of their neural architecture designs or probabilistic estimation strategies.

Response to Question 1

Thank you for bringing up these forward-thinking questions. We concur that zero-shot/few-shot/transfer/pretraining are all valuable and under-explored paradigms in the realm of time-series forecasting.

As stated in our paper, our focus is on unifying probabilistic and non-probabilistic methods and facilitating comprehensive evaluations across diverse data characteristics. In doing so, we highlight many overlooked challenges, unresolved issues, and the strengths and weaknesses of existing approaches. Hence, we chose to concentrate on these aspects within this paper.

Looking ahead, we believe that revealing these unique insights to the community will also be instrumental in advancing research in time-series forecasting within the zero-shot/few-shot/transfer/pre-training paradigms. This is because all these learning paradigms involve the selection of probabilistic or non-probabilistic estimations, custom or general architectures, and autoregressive or non-autoregressive decoding schemes. As for ProbTS itself, we believe it is certainly feasible to extend its scope beyond the supervised paradigm of time-series forecasting. This is due to the fact that we have compiled a wide range of datasets and aim to unify neural architecture designs and probabilistic estimation abilities. Based on your suggestion, we will include a discussion on these aspects in the revised paper, thus presenting a broader vision and greater potential impacts.

Response to Question 2

Currently, we have not encountered issues with RAM insufficiency. To facilitate machine learning on large quantities of data that do not fit into RAM, it's essential to establish a comprehensive data pipeline that systematically manages data sharding, indexing, shuffling, queuing, and so forth. For this purpose, we could learn from successful toolkits in language and vision domains. However, it's important to note that the focus of this paper is on unifying different research branches in time-series forecasting, rather than designing efficient pipelines to handle large-scale time-series data.

Regarding your query about licenses, rest assured that all datasets utilized in our experiments are open-sourced and publicly accessible. We've compiled the sources of these datasets in the table below for your reference.

Response to Weaknesses 2

Thank you for your suggestion to incorporate the M4, M5, and TOURISM datasets, which are indeed key datasets for univariate time-series forecasting. In light of your advice, we have analyzed these datasets and conducted experiments comparing different forecasting methods.

Our findings indicate that the characteristics of these new datasets do not extend beyond the scope of the existing datasets used in this paper. Furthermore, the additional tests run on these new datasets further reinforce the major findings and conclusions of our paper. We provide more detailed information about these two points below for your convenience.

Table 1 presents the comparison of the new datasets (M4, M5, TOURISM) with the existing datasets. As per the data analysis strategy detailed in our paper, we focus on three key aspects: the strength of trending, the strength of seasonality, and the complexity of data distribution. From the table, it is evident that these new datasets do not introduce particularly unique characteristics, with the exception of M4-Daily, which may exhibit fewer seasonal patterns. The main difference between the new datasets and the existing ones is that the new datasets are all designed for univariate time-series forecasting, while the datasets in our study primarily target multivariate time-series forecasting. Thanks to your constructive suggestion, we now have a broader spectrum covering both univariate and multivariate scenarios, thereby reinforcing our findings.

Table 1. Quantitative assessment of the intrinsic characteristics of the datasets reported in our paper. The JS Div. denotes Jensen-Shannon divergence, where a lower score indicates closer approximations to a Gaussian distribution.

Table 2. Quantitative assessment of the intrinsic characteristics of the univariate datasets.

Table 3 includes the experimental results of some representative methods. These findings align with our initial observations discussed in the paper. For example, methods that specialize in probabilistic estimation, such as GRU NVP and TimeGrad, demonstrate superior performance on datasets with complex distributions like M4-Weekly and M5. On the other hand, on the TOURISM-Monthly dataset, which has a relatively simple data distribution, baseline methods equipped with a simple point forecaster, like DLinear and PatchTST, also perform well. Additionally, both autoregressive and non-autoregressive decoding schemes show comparable performance in short-term forecasting.

Table 3. Results on M4, M5, and TOURISM datasets. We utilize a lookback window of 3H, with 'H' denoting the forecasting horizon.

Thank you for your thorough review. We are particularly grateful for your recognition of the key strengths of our paper, specifically our unification of probabilistic and non-probabilistic methods to reveal novel insights, identify current challenges, and suggest potential directions for future research. In response to your additional queries, we trust that the following explanations will satisfactorily address your concerns.

Response to Weaknesses 1

Thank you for this insightful suggestion. As per your recommendation, we have incorporated a few classical statistical methods such as ARIMA and ETS into the ProbTS framework. This addition serves to further solidify and enrich our empirical research. We invite you to review the table below which compares these baseline methods with state-of-the-art deep learning techniques.

As can be observed, deep learning methods significantly outperform these simple statistical baselines. This highlights the crucial role of capturing non-linear dependencies in generating accurate forecasts. Such comparisons underscore the necessity of further research to enhance deep time-series forecasting, which is one of the primary goals of our paper. Additionally, we have noted that ARIMA and ETS have distinct performance tendencies depending on specific data characteristics. For instance, the ARIMA method struggles with datasets like 'Solar' which exhibit weak trending and strong seasonality, whereas the ETS model shows better adaptability. Conversely, in cases of strong trending and weak seasonality, as seen in the 'Wikipedia' dataset, the ARIMA model significantly outperforms the ETS model.

[1] Asteriou, Dimitros; Hall, Stephen G. (2011). "ARIMA Models and the Box-Jenkins Methodology". Applied Econometrics (Seconded.). Palgrave MacMillan. pp.265-286.ISBN978-0-230-27182-1.

[2] Hyndman, Rob J., and Athanasopoulos, George. Forecasting: principles and practice, 3rd edition, OTexts, 2021.https://otexts.com/fpp3/expsmooth.html

[3] https://www.statsmodels.org/dev/examples/notebooks/generated/ets.html