MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process

We sincerely appreciate your thorough review and valuable feedback. Enclosed are our responses to your insightful questions and concerns.

Q1. According to the inference procedure, the proposed time series forecasting approach can predict one future horizon time step at a time. How to apply this predictive approach for the long-term time series forecasting task effectively?

We are grateful to the reviewer for the insights.

• In our Temporal Process Module, we utilize the RNN as the backbone model to encode the temporal dependence sequentially. Therefore, predicting m prediction length involves taking n-to-1 (n timesteps predicts one timestep) for m times autoregressively, where each inference step incorporates previous predictions into the context. Based on the extra experiment concerning long-term forecasting, our method experiences only a moderate decrease in performance while extending prediction length.
• It is worth noting that our Temporal Process Module is adaptable and can accommodate other encoders capable of handling variable-length time steps. This includes, but is not limited to, autoregressive RNNs in an n-to-1 manner or non-autoregressive methods of an n-to-m manner such as transformers.
• In certain scenarios, a Transformer-based TPM might be a more efficient solution, taking m timesteps predictions with one step. However, Transformers have limitations regarding the maximum length of the time series, which means the context length and prediction length cannot be increased infinitely. Additionally, for high-dimensional time series data (such as Wikipedia dataset with 2000 dimensions), the attention mechanism might suffer from huge training and inference overhead.

Since the primary focus of this paper is on multi-granularity guidance, we did not test the eligibility of transformer variant for TPM and we will take it as an important future work.

W1. The assumption that "... forward process of the diffusion model ... intuitively aligns with the process of smoothing fine-grained data into a coarser-grained representation ..." is not verified through theoretical analysis or empirical study.

Thanks to the reviewer for the valuable insights. We would like to point out that the results depicted in Figure 3 of our manuscript provide empirical support for our assumption. Please note that, in Figure 3, the x-axis denotes the denoising diffusion steps, and the y-axis signifies the $\\text{CRPS}\_{\\text{sum}}$ values.

Figure 3 illustrated an investigation into the effect of share ratio, where we evaluate the performance of MG-TSD using various share ratios across different coarse granularities.

In Figure 3(a)-(d), the dashed blue curve in each plot represent $\\text{CRPS}\_{\\text{sum}}$ values between the coarse-grained targets and 1-hour samples come from 1-gran (finest-gran) model at each intermediate denoising step; each point on the orange polylines represents the $\\text{CRPS}\_{\\text{sum}}$ value of 1-hour predictions by 2-gran MG-TSD models (where one coarse granularity is utilized to guide the learning process for the finest-grained data), with different share ratios ranging from [0.2, 0.4, 0.6, 0.8, 1.0], and the lowest point of the line segment can be used to characterize the most suitable share ratio for the corresponding granularity.

The four subplots from (a) to (d) illustrate a gradual smoothing transformation of the distribution of increasingly coarser targets. A key observation is that from the left to the right panel, the distribution of coarser targets gradually aligns with the distribution of intermediate samples at larger diffusion steps. More specifically, as granularity transitions from fine to coarse (4h→6h→12h→24h), the denoising steps at which the distribution most resembles the coarse-grained targets increase (approximately at steps 20→40→60→60). This comparison underscores the similarity between the diffusion process and the smoothing process from the finest-grained data to coarse-grained data, both of which involve a gradual loss of finer characteristics from the finest-grained data through a smooth transformation.

W2. Experimental settings for the time series forecasting task are unclear.

Thanks the reviewer for the question. To clarify, we generate coarse-grained data from the finest-grained time series data over the entire timeline. We rely on the GluonTS[1] library for data splitting and creating the training and testing instances. The GluonTS library is widely used in time series forecasting. Further explanations are provided below:

• Generation of the coarse-grained data: Section 3.1 has the details to generate multi-granularity data. A sliding window with a pre-defined size $s^g$ for granularity $g$ is applied to the finest-grained data across the entire timeline, where $g=1,2,...,G$. These sliding windows are intentionally non-overlapping. Within each window, we smooth the finest-grained data by averaging and replicate the average $s^g$ times to align with the timeline $[1, T]$.
• Dataset splitting: All the datasets we used in the benchmark experiments are public and available in GluonTS[1]. The library has these datasets pre-split into training and testing sets, and we follow this splitting method to obtain our training and testing datasets.
• Creation of training and testing instances: we randomly sample the context window, followed by the prediction window, from the complete training data. This process can be viewed as applying a moving window to auto-regressively roll through the entire timeline, with consecutive time intervals overlapping. Furthermore, for different datasets, the context length and prediction length vary, such as 24-hour-24-hour for Electricity, and 30-Day-30-Day for Wikipedia, as detailed in Appendix C.1.

W3. More experimental results about different prediction lengths are expected.

Thanks the reviewer for insightful suggestion. We have conducted more experiments to test the performance of MG-TSD method with different prediction length.

Experiment setting: In the current experiment, we assess the long-term prediction performance of MG-TSD by comparing it with the baseline TimeGrad, chosen for its competitive performance in the main experiment (Table 1) and its shared characteristic of forecasting in an autoregressive manner. For the solar and electricity datasets (with the frequency of 1 hour), we set the context length to 24 hours and evaluate the performance of methods with prediction lengths of 24 hours, 48 hours, 96 hours and 144 hours. The average $\\text{CRPS}\_{\\text{sum}}$, $\\text{NRMSE}\_{\\text{sum}}$, and $\\text{NMAE}\_{\\text{sum}}$ metrics are computed for both MG-TSD and the baseline over 10 independent runs, with error bars indicating the corresponding standard deviations. (We would include additional baselines in the plots later to strengthen our conclusions for updated paper. )

Experiment results and findings: The preliminary results indicate that MG-TSD performs well for long-time forecasting. The results indicate that as the prediction length increases, the performance of our proposed method stays robust, exhibiting no sudden decline. Furthermore, our method consistently outperforms the competitive baseline. This performance advantage is anticipated to persist in future trends, with no indication of convergence between the approaches.

Table G: Results for Solar Dataset

Table H: Results for Electricity Dataset

Action taken: We have included the experiments and results in Appendix B.2. Figure 5 in the appendix visualizes the results in Table G and Table H provided here, providing a clearer presentation of our conclusions.

W4. Reproducibility

We plan to release our code soon. We are intensively working on cleaning up our code to ensure it is well-documented, thoroughly tested, and user-friendly. Additionally, to guarantee the reproducibility of our experiments, we performed 10 independent runs for each setting and reported both the average values and the standard deviation (std).

References:

[1]Alexandrov A, Benidis K, Bohlke-Schneider M, et al. Gluonts: Probabilistic and neural time series modeling in python[J]. The Journal of Machine Learning Research, 2020, 21(1): 4629-4634.

W2. Concern about the ablation study for the Multi-granularity Data Generator module and TPM module

Thanks to the reviewer for the comment.

We would like to highlight that Table 3 in our manuscript serves as an ablation study of the Multi-granularity component in the MG-TSD model. The experiment, presented in Table 3, explores the effect of varying the number of granularity levels on the model's performance. Testing the performance of the model with an additional granularity level inherently assesses the effectiveness of all three modules. More specifically, the Multi-granularity Data Generator Module is executed to generate the data at a new granularity level. The Temporal Process Module (TPM) incorporates an additional RNN cell to encode the temporal pattern within the added granularity data, while the Guided Diffusion Process Module adapts the learning and generation process with the additional coarse-grained data. The results in Table 3 reveal that increasing the number of granularity levels typically improves the performance of the MG-TSD model. This finding confirms that the mechanism currently designed to generate coarse-grained data is effective in enhancing performance. Theoretically, the MG-TSD model can handle an unlimited increase in granularity levels. However, in practical scenarios, we observe that the marginal benefit tends to diminish as the number of granularity levels increases. Our findings suggest that employing four to five granularity levels typically suffices to achieve optimal performance.

To further clarify, the Multi-granularity Generator Module functions as a predefined data pre-processing step and is utilized to generate coarse-grained data at various granularity levels. It does not involve parameter optimization during the training stage. The Temporal Process Module serves as an encoder in the model, which captures and compresses the temporal dependencies in time series data up to a certain timestep. The RNN is adopted in our model as it presents a feasible and convenient method of implementation for this module. The choice of RNN as the backbone encoder in TPM aligns with various previous works, including TimeGrad[1] and GP-copula[2]. These studies utilize RNN for modeling temporal dependencies, which attests to the effectiveness of RNN. Since the primary focus of this paper is on multi-granularity guidance, we did not conduct extra ablation studies on the designs of these components.

References:

[1] Rasul K, Seward C, Schuster I, et al. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting[C]//International Conference on Machine Learning. PMLR, 2021: 8857-8868.

[2] Salinas D, Bohlke-Schneider M, Callot L, et al. High-dimensional multivariate forecasting with low-rank Gaussian copula processes[J]. Advances in neural information processing systems, 2019, 32.

We are grateful for your constructive comments. Below we address each question and concern.

Q1: In Equation 7 of Section 3.2.1, is the distribution of $\\epsilon$ consistent with the distribution of $x\_N$? Does the distribution of $\\epsilon$ obey the normal distribution? The distribution and meaning of $\\epsilon$ are not pointed out.

Thank the reviewer for bringing attention to this. The $\\epsilon$ in equation (7) adheres to the standard normal distribution, consistent with the backbone denoising diffusion models. The dimension of the $\\epsilon$ is the same with $x^{g}$.

Action taken: We have incorporated this clarification into our manuscript.

Q2: Compared with the existing diffusion probabilistic models, does the MG-TSD model framework differs only in the Guided Diffusion Process Module?

Thanks for this question. We would like to further clarify on our module design.

• The core difference between MG-TSD and previous work principally lies in the approach of incorporating information from data. We creatively propose a multi-granularity guidance approach to naturally exploit intrinsic coarse-to-grain features within data to stabilize the samples from the diffusion model. This is facilitated by the Guided Diffusion Process Module.
• The multi-granularity data generator is unique in our work for the preprocessing of original data into multiple alternatives with different granularities. This module collaborates synergistically with the Guided Diffusion Process Module, enabling it to adapt to multi-granularity requirements.
• Although the temporal process module is similar to previous work[1][2] in its function for extracting historical context, it was dedicatedly designed to adapt for the multi-granularity cases. To be more specific, temporal process module employs separate RNN submodule for each granularity, without parameters sharing across different granularities.

W1. Concern about the time complexity and memory of the model.

Experiments have been conducted to evaluate the time and memory usage of the MG-TSD model during training across various granularities.

Experiment setting: These experiments were executed using a single A6000 card with 48G memory capacity. The Solar dataset was utilized in this context, with a batch size of 128, input size of 552, 100 diffusion steps, and 30 epochs.

Experiment results and findings: As illustrated in the corresponding graph and table, there is a linear increase in memory consumption with an increase in granularity. A slight addition in training time is also observed. These findings are coherent with the architecture of our model. In particular, each additional granularity results in the introduction of an extra RNN in the Temporal Process Module and an increase in computation within the Guided Diffusion Process Module. As per theoretical expectations, these resource consumptions should exhibit linear growth. Moreover, it is pertinent to mention that an excessive increase in granularity may not notably boost the final prediction results, hence the granularity should be kept within a certain range. Therefore, the consumption of memory will not rise indefinitely.

Table F: Comparison of Time and Memory Consumption at Different Granularity Levels in MG-TSD Model Training

Summary: In summary, while it is evident that an increase in granularity escalates the consumption of both time and memory, these increases are reasonable and fall within acceptable boundaries.

Action taken: We have included the experiments and results in Appendix B.2. Figure 6 in the appendix visualizes the results in Table F provided here, providing a clearer presentation of our conclusions.

We provide more statistics summaries to evaluate the performance improvement. We have recorded the rankings of different methods based on their performance on various datasets. The results are shown in table below.

Table D: Performance ranking of methods across datasets

From the results, our method achieved the best rank among all baselines on six datasets in terms of the $\\text{CRPS}\_{\\text{sum}}$. As for the $\\text{NRMSE}\_{\\text{sum}}$ and $\\text{NMAE}\_{\\text{sum}}$ metrics, we achieved the best performance on all datasets, except for KDD-cup dataset where the GP-Copular method was slightly better.

Additionally, we have recorded the percentage improvement of the MG-TSD method relative to the top-performing baseline in the table below for $\\text{CRPS}\_{\\text{sum}}$.

Table E: Relative improvement of MG-TSD method over top-performing baseline

On the six datasets, we achieved $\\text{CRPS}\_{\\text{sum}}$ improvements, ranging from 4.7% to 35.8%. The modest improvements were achieved on the Wikipedia dataset. This might be due to its high-dimensional nature. The improvement of our model is not trivial, given the challenges posed by these datasets.

Action Taken: In Tables 1, 4, and 5, we have updated the results to reflect the best performance of the MG-TSD model with multiple granularities, rather than just two.

W4. The use of the RNN architecture requires further explanation.

Thanks for the question. To clarify, the Temporal Process Module serves as an encoder in the model, which captures and compresses the temporal dependencies in time series data up to a certain timestep. The Recurrent Neural Network (RNN) is adopted in our model as it presents a feasible and convenient method of implementation for this module. It is worth noting that our Temporal Process Module is adaptable and can accommodate other encoders capable of handling variable-length time steps. This includes, but is not limited to, autoregressive RNNs in an n-to-1 manner or transformers in an n-to-m manner.

Furthermore, the choice of RNN as the backbone encoder in TPM aligns with various previous works, including TimeGrad[8] and GP-copula[9]. These studies utilize RNN for modeling temporal dependencies, which attests to the effectiveness of RNN. Notably, differing from these existing works, which typically use a single RNN for single-granularity data, we employ multiple RNNs in the model to leverage the pattern and temporal information in time series data at various granularity levels. These RNNs operate without parameter sharing and are trained simultaneously with the Guided Diffusion Process Module.

References:

[1]Bjerregård M B, Møller J K, Madsen H. An introduction to multivariate probabilistic forecast evaluation[J]. Energy and AI, 2021, 4: 100058.

[2]Rasul K, Sheikh A S, Schuster I, et al. Multivariate probabilistic time series forecasting via conditioned normalizing flows[J]. arXiv preprint arXiv:2002.06103, 2020.

[3]Lin L, Li Z, Li R, et al. Diffusion models for time series applications: A survey[J]. arXiv preprint arXiv:2305.00624, 2023.

[4]Shen L, Kwok J. Non-autoregressive Conditional Diffusion Models for Time Series Prediction[J]. arXiv preprint arXiv:2306.05043, 2023.

[5] Li Y, Lu X, Wang Y, et al. Generative time series forecasting with diffusion, denoise, and disentanglement[J]. Advances in Neural Information Processing Systems, 2022, 35: 23009-23022.

[6] Nie Y, Nguyen N H, Sinthong P, et al. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers[C]//The Eleventh International Conference on Learning Representations. 2022.

[7] Wu H, Xu J, Wang J, et al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting[J]. Advances in Neural Information Processing Systems, 2021, 34: 22419-22430.

[8] Rasul K, Seward C, Schuster I, et al. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting[C]//International Conference on Machine Learning. PMLR, 2021: 8857-8868.

[9] Salinas D, Bohlke-Schneider M, Callot L, et al. High-dimensional multivariate forecasting with low-rank gaussian copula processes[J]. Advances in neural information processing systems, 2019, 32.

Table A: Comparison of $\\text{CRPS}\_{\\text{sum}}$ (smaller is better) of models on six real-world datasets. The reported mean and standard error are obtained from 10 re-training and evaluation independent runs.

Table B: Comparison of $\\text{NRMSE}\_{\\text{sum}}$ (smaller is better) of models on six real-world datasets. The reported mean and standard error are obtained from 10 re-training and evaluation independent runs.

Table C: Comparison of $\\text{NMAE}\_{\\text{sum}}$ (smaller is better) of models on six real-world datasets. The reported mean and standard error are obtained from 10 re-training and evaluation independent runs.

W3. Concerns about the significance of the model’s improvements compared to baselines.

For clarity, the original Table 1 results illustrate the MG-TSD model's performance with two granularities. Our results in Table 3 suggests employing more granularity levels could lead to further performance improvements. We have updated results in Tables 1, 4 and 5 to better reflect the MG-TSD model's optimal performance using multiple granularities.

Thank you very much for your careful review and constructive suggestions! Please find our response to your questions and concerns.

Q1&W1. Some related works can be further discussed. There have been some similar works, such as TimeDiff[1] and D3VAE[2], which also apply the diffusion model. What are the technical advantages of these studies?

Diffusion-based methods that use generations from conditional distribution as predictions are typically from probabilistic methods. In contrast, sequence-based (including transformer-based) models are mostly deterministic methods.

Autoformer and PatchTST are deterministic time-series forecasting models based on dedicated designed Transformers, which are different from MG-TSD in the scope of prediction. D3VAE and TimeDiff are probabilistic time-series forecasting methods, which are closer to the scope of our work.

Both D3VAE and TimeDiff involve diffusion probabilistic models. D3VAE utilizes only the forward diffusion process and the prediction stage is majorly taken over by a VAE architecture, which is fundamentally different from MG-TSD in the way of generating forecasts.

In contrast, both TimeDiff and MG-TSD utilize a conditional denoising process over time intervals to perform forecasting. The major difference between TimeDiff and MG-TSD lies in the way of incorporating information within data, TimeDiff is trained with the mixed up of hidden contexts and future ground truths as sample conditionings, while MG-TSD utilizes multi-granularity guidance, leveraging intrinsic coarse-to-grain features within data.

The technical advantages of diffusion-based time series forecasting are summarized as below:

• Uncertainty quantification. A primary advantage of probabilistic models is their capability to accurately capture data distribution instead of just point estimates. This allows for the convenient construction of prediction intervals using multiple outputs from the diffusion model. By knowing the data distribution at a specific timestamp, one can quantify the prediction uncertainty and provide more reliable forecasts. It also enables the convenient evaluation of the probability of extreme events occurring. For instance, in the wind power sector, reliable forecasts are essential. An unexpected extreme event causing a wind farm to shut down can wipe out months of revenue. This underscores the importance of probabilistic forecasting, which considers both the expected power output and the uncertainty of the forecast, thereby aiding in the minimization of such risks [1].

• The capability to model arbitrary distributions without parametric assumptions. This capacity of diffusion models to characterize distributions is also applicable to modeling the distribution of time series. A flaw in other methods of distribution modeling is that they are strictly constrained by the functional structure of their target distributions. For example, the previous choice, Transformer-MAF[2], models multivariate time series with an autoregressive deep learning model, in which the data distribution is expressed by a conditional normalizing flow. Diffusion-based methods, conversely, can offer a less restrictive solution, as indicated in reference [3].

Action Taken: We have included additional experimental results with respect to these mentioned works, on datasets of different dimensions and predicting lengths. Overall, the performance of MG-TSD is superior. Please refer to the reply to Q2&W2 below for numerical results. We have also included these results in the updated manuscript.

Q2&W2. The paper designed MG-TSD based on the diffusion model. Why is it not compared with the diffusion-based models in the baseline? Besides, there are some newer Transformer-based models, such as PatchTST[6], and Autoformer[7] should be compared in your experiments. There is only one metric in the main experiment, which is not enough.

Thanks for suggesting additional references. In our original benchmark experiment, the TimeGrad method included in our baseline is a diffusion-based model. To ensure comprehensive comparisons, we have now incorporated all the mentioned works, TimeDiff[4], D3VAE[5], PatchTST[6], and AutoForemer[7], into our baselines.

Furthermore, we have broadened our evaluation metrics to include $\\text{NMAE}\_{\\text{sum}}$ (Normalized MAE) and $\\text{NRMSE}\_{\\text{sum}}$ (Normalized RMSE). Detailed results for these additional metrics can now be found in the updated appendix, particularly in Tables 4 and 5.

Here, we attach the benchmark experiment results of three metrics $\\text{CRPS}\_{\\text{sum}}$, $\\text{NMAE}\_{\\text{sum}}$, and $\\text{NRMSE}\_{\\text{sum}}$ below. These additional experiments validate that our method consistently delivers state-of-the-art performance.