Multi-Resolution Diffusion Models for Time Series Forecasting

Q1: It appears that the networks at different stages are independent, which suggests that the total number of parameters in the proposed model is significantly higher than the ones in other models. If so, the performance of mr-diff would be less impressive than what appears in the paper. I think the authors should discuss this issue in the paper to avoid potential misinterpretation.

A1: Indeed, the number of parameters in mr-Diff is often smaller than that in other diffusion models. The table below compares the number of trainable parameters for the various time series diffusion models. As can be seen, the number of parameters in mr-Diff is smaller than those in TimeGrad, CSDI and SSSD. This is because mr-Diff uses convolution layers in its denoising networks, which eliminates the need for large modules like residual blocks (TimeGrad and SSSD) or self-attention layers (CSDI).

Q2: More details should be given regarding the seasonal-trend decomposition and the generation of the coarse trends.

A2: The seasonal-trend decomposition is a classic time series method that aims to extract informative trend and seasonal components from time series [1]. The decomposition method utilized in this paper is described by the trend extraction equation presented in Section 4.1, which corresponds to the decomposition method described in [2]. At each decomposition stage, it takes a finer trend (such as $\mathbf{X}_1$ and $\mathbf{Y}_1$ in Figure 2) as the input and generates a coarser trend (such as $\mathbf{X}_2$ and $\mathbf{Y}_2$ in Figure 2).

References:

• [1] Cleveland R B, Cleveland W S, McRae J E, et al. STL: A seasonal-trend decomposition. J. Off. Stat, 1990, 6(1): 3-73.

• [2] NeurIPS'21 Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.

Q3: Additionally, the paper claims that the proposed model enjoys a lower computational complexity by dropping the attention layer in their architecture; however, this may result that when the generated time series is sufficiently long, the beginning and the end of the series might be independent of each other and cause some inconsistency problem. I wonder if the authors could give some comments on this issue.

A3: This problem is not observed in our long-term forecasting experiments (e.g., with prediction horizon $H = 720$ on NorPool and Caiso and $H = 168$ for Traffic and Electricity). These $H$ values are usually considered long-term forecasting in the literature. As can be seen from Tables 1-2 and 13-14, mr-Diff consistently achieves the highest average ranking. This demonstrates the effectiveness of mr-Diff for long-term forecasting. This is mainly because the proposed mr-Diff combines the merits of diffusion model and multiresolution analysis. The diffusion model provides a strong ability for data generation and multiresolution analysis allows the use of coarser trends to help capture long-term dependencies.

Q1: More analysis is needed on the experiment results. The experiment results seem to suggest that the performance does not benefit a lot from the multi-resolution analysis. The performance gain is not significant after comparing it with the best models in diffusion models, basis expansion-based models, and the last category of models. The authors claimed that the lack of complex trend information is responsible for the absence of improvement. Could the authors provide further elaboration on this point? What could be considered as complex trends that the proposed model can take advantage of?

A1: We respectfully disagree with the statement that "The performance gain is not significant after comparing it with the best models in diffusion models, basis expansion-based models, and the last category of models''. It is difficult for a method to outperform all baselines on all datasets, and the ranking of mr-Diff is higher than all other baselines.

Specifically, in Table 1, mr-Diff is the best in 5 of the 9 datasets and the second best in 3 datasets. In comparison, the best diffusion model baseline is TimeDiff, which ranks second best on only one dataset. The best expansion-based model, FiLM, achieves the best on only one dataset. The best model in the last category, NLinear, is the best on two datasets and the second best in two datasets.

We mentioned in Section 5.1 that there is no improvement on datasets such as Caiso. This is because there is no complex trend information on this dataset that can be leveraged (as shown in Figure 5 (b) in Appendix B). As can be seen from Figure 5 (b), the Caiso data is relatively simple as it presents clear periodicity. Hence, on this easy dataset, the improvement is not very significant. However, the dataset ETTh1 (Figure 4) is more challenging as it exhibits complex temporal behaviors (such as nonstationarity, nonlinear temporal patterns, irregular oscillations, and abrupt changes) that are not easily captured by simple or linear trend models. By combining the merits of diffusion models and multiresolution analysis, the proposed mr-Diff achieves better results in this data.

Q2: What is the inference time of the proposed model compared to NLinear and DLinear?

A2: As suggested, we include the inference time (in ms) of NLinear and DLinear at different prediction horizons ($H$) in the table below. Note that NLinear and DLinear are linear models, and thus have faster training and inference compared to the proposed model and other deep learning methods. However, their performance are limited on the more complicated datasets. For example, as can be seen from Table 1, NLinear ranks 6 and 4 on the NorPool and ETTh1 datasets, respectively; whereas the proposed mr-Diff ranks 2 on both datasets. Compared to the other time series diffusion models, it is important to note that the proposed mr-Diff model is much faster than TimeGrad, CSDI and SSSD, and is faster or comparable with TimeDiff depending on the setting of $S$.

Q3: The performance gain in multivariate prediction is even less significant when comparing the data in Tables 1 and 2, as well as in Tables 13 and 14. Can the authors elaborate on this observation?

A3: We respectfully disagree with that "The performance gain in multivariate prediction is even less significant". The results in Tables 1-2 and 13-14 show that the proposed mr-Diff consistently ranks higher than the other baselines across all nine datasets. In particular, for multivariate prediction, it is best on 1 dataset and second-best on 4 of 9 datasets (Table 2). In Table 14, its MSE is the best on 3 of 9 datasets and second-best on 2 datasets.

Q1: The proposed model is restricted to/evaluated on only forecasting tasks but not on generation tasks and downstream tasks like classification (c.f. https://arxiv.org/abs/2303.09489)

A1: This paper focuses on time series forecasting by incorporating multiresolution analysis into time series diffusion models. A number of popular related papers [1-5] also focus only on time series forecasting. Moreover, as mentioned in Section 3.2, time series forecasting requires predicting future values conditioned on past observations, and thus forecasting can also be regarded as generation. Extending the proposed diffusion model to classification is an interesting topic, and will be studied in the future.

References:

• [1] ICML'23 Non-autoregressive conditional diffusion models for time series prediction.

• [2] ICLR'23 Scaleformer: Iterative multi-scale refining transformers for time series forecasting.

• [3] ICLR'22 DEPTS: Deep expansion learning for periodic time series forecasting.

• [4] NeurIPS'22 Generative time series forecasting with diffusion, denoise, and disentanglement.

• [5] ICML'22 FEDformer: frequency enhanced decomposed transformer for long-term series forecasting.

Q2: The proposed model is somewhat complicated, consisting of many components, and it is not clear how the model performance would be affected if one of the components is not used, as well as how this depends on the multiscale nature of the time series. A systematic approach is to carefully evaluate the model on synthetic time series data generated by multiscale dynamical systems; e.g., multiscale Lorenz-96 systems

A2: There might be some misunderstanding. Indeed, as mentioned in the first paragraph of Section 5, we have provided further studies on hyperparameters/components in the proposed mr-Diff. However, because of the lack of space, these ablation studies are reported in Appendix F. In particular, we have studied three important components: i) replacing the seasonal-trend decomposition by downsampling/upsampling operations, ii) varying the number of decomposition stages $S$ (note that using $S=1$ is the same as not performing seasonal-trend decomposition), and iii) removing future mixup. The results of these studies demonstrate the effectiveness of these components in the proposed diffusion model.

As suggested by the reviewer, we evaluate the proposed model on the multiscale Lorenz-96 systems. The prediction length $H$ is set to 96. The MAE results are shown in the table below. As can be seen, mr-Diff consistently outperforms other baselines, as it combines the merits of diffusion models and multiresolution analysis.

Q3: Missing discussions on training efficiency and number of trainable parameters, and how do they compare to other models

A3: As suggested by the reviewer, the table below shows the number of trainable parameters and the training time (in ms) of different time series diffusion models. As can be seen, the number of trainable parameters is smaller than those of TimeGrad, CSDI and SSSD. This is because mr-Diff uses convolution layers in its denoising networks, which eliminates the need for large modules like residual blocks (in TimeGrad and SSSD) and self-attention layers (iin CSDI). Moreover, mr-Diff is also faster than TimeDiff, thanks to its use of a linear mapping for future mixup instead of employing additional deep layers for this purpose.

Q4: Missing evaluations on other forecasting tasks in the Monash benchmark. How will the proposed model perform on all the tasks of the Monash benchmark?

A4: As suggested by the reviewer, we add experiments on the Monash benchmark. Because of the limited time of the rebuttal period, only 12 Monash datasets are used. Note that the submission already includes nine real-world datasets that exhibit diverse temporal characteristics (such as different stationary states, periodicities, and sampling rates) and have been commonly used in related time series prediction works (including Autoformer (NeurIPS'21), SCINet (NeurIPS'22), FiLM (NeurIPS'22), and Depts (ICLR'22)). MAE results are shown in the table below. As can be seen, mr-Diff continues to outperform other baselines. It achieves the best results in 10 out of the 12 datasets, and the second-best in the remaining 2 datasets.

Q5: Error bars evaluation is not completed (provided for only Electricity, ETTh1, ETTm1 but not the others

A5: As suggested by the reviewer, error bars for the other five datasets are provided in the table below. As can be seen, the largest standard deviation is 0.0023, indicating that the proposed model is robust to different initializations.

Q6: MAE and MSE are used as metrics to evaluate the forecasting performance. How about using MAPE?

A6: It is known that MAPE has limitations [1-2]. One of its main problems is that it is meaningful only for values where divisions and ratios make sense, and produces infinite or undefined values when the actual observations are zero or close to zero. Moreover, it can be biased by the scale of the actual values. For example, suppose you have two observations with the same absolute error, but different actual values. In that case, the one with the smaller actual value will have a larger percentage error and contribute more to the MAPE. Hence, most related time series papers (such as [3-6] below) do not report MAPE.

References:

• [1] Tayman J, Swanson D A. On the validity of MAPE as a measure of population forecast accuracy. Population Research and Policy Review, 1999, 18: 299-322.

• [2] Kim S, Kim H. A new metric of absolute percentage error for intermittent demand forecasts. International Journal of Forecasting, 2016, 32(3): 669-679.

• [3] ICML'23 Non-autoregressive conditional diffusion models for time series prediction.

• [4] ICLR'23 Scaleformer: Iterative multi-scale refining transformers for time series forecasting.

• [5] NeurIPS'22 Generative time series forecasting with diffusion, denoise, and disentanglement.

• [6] ICML'22 FEDformer: frequency enhanced decomposed transformer for long-term series forecasting.

Q7: How effective is the model for long-term forecasting (i.e. when H is large)? Can it reconstruct the long-term statistics of the underlying dynamical system accurately?

A7: Many of our experiments are on long-term forecasting, e.g., with prediction horizon $H=720$ (corresponding to forecasting next month) on NorPool and Caiso, and $H=168$ for Traffic and Electricity (corresponding to forecasting next week). These $H$ values are usually considered long-term forecasting (see, e.g., [1-3]). As can be seen from Tables 1 and 2, mr-Diff consistently achieves the best average ranking. This demonstrates the effectiveness of mr-Diff for long-term forecasting.

References:

• [1] ICML'23 Non-autoregressive conditional diffusion models for time series prediction.

• [2] ICLR'23 Scaleformer: Iterative multi-scale refining transformers for time series forecasting.

• [3] ICML'22 FEDformer: frequency-enhanced decomposed transformer for long-term series forecasting.

Q8: What are the limitations of the proposed model?

A8: The main limitation is that the lookback window needs to be sufficiently long and informative. When the lookback window is too short, extracting multiresolution trends becomes challenging, and performance improvement can be less obvious.

Thank you for the clarifications and the additional results. I am raising my score. Please also include the source code that could reproduce the results in the next revised version.

Q1: I would be happy if these discussions and results were included in a revised version. I am keeping my score.

A1: We would like to sincerely thank Reviewer V8ZH for providing valuable suggestions.

As suggested by the reviewer, we have included the above discussions and results in the revised version.

The main changes in the revised version include:

• Inclusion of Appendix G, which reports results on the multiscale Lorenz-96 time series for different H's.
• Addition of Appendix H, which presents the Monash results.
• Introduction of Tables 12 and 13 in Appendix I, providing a complete error-bar evaluation.
• Incorporation of Appendix J, which includes discussions regarding the training efficiency and the number of trainable parameters of the proposed model.

Q2: For the multiscale Lorenz-96 system, what is the precise equation (and parameters) used for generating the synthetic data and what is the dimension of the system?

A2: The precise equation of the multi-scale Lorenz 96 system is $\frac{dX_k}{dt}=X_{k-1}\left(X_{k+1}-X_{k-2}\right)+F-\frac{hc}{b}\Sigma_jY_{j,k},$ $\frac{dY_{j,k}}{dt}=-cbY_{j+1,k}\left(Y_{j+2,k}-Y_{j-1,k}\right)-cY_{j,k}+ \frac{hc}{b}X_k -\frac{he}{d}\Sigma_iZ_{i,j,k},$ $\frac{dZ_{i,j,k}}{dt}=edZ_{i-1,j,k}\left(Z_{i+1,j,k}-Z_{i-2,j,k}\right)- geZ_{i,j,k}+ \frac{he}{d}Y_{j,k}.$ Following [1], the values of the parameters are set as follows: $F=20$, $b=c=e=d=g=10$, and $h=1$. The indices $i, j, k$ range from $1$ to $8$, resulting in $X$ having 8 dimensions, while $Y$ and $Z$ have 64 and 512 dimensions, respectively. We utilize the time series data of variable $X$ in the experiments.

Reference:

[1] Chattopadhyay A, Subel A, Hassanzadeh P. Data-driven super-parameterization using deep learning: Experimentation with multiscale Lorenz 96 systems and transfer learning[J]. Journal of Advances in Modeling Earth Systems, 2020, 12(11): e2020MS002084.

Q3: For the evaluation, it would also be nice to see how various models perform for different values of H (from small to large).

A3: The table below summarizes results for different horizons ($H$ in {24, 96, 168, 336, 720}). As can be seen, mr-Diff consistently outperforms other baselines across different values of $H$.

Q1: The authors mentioned that there have been recent works using U-Net to capture multi-resolution patterns. Considering that the main contribution of this work compared with the TimeDiff is incorporating a multi-resolution structure in a diffusion model, it is essential to compare the proposed seasonal-trend decomposition and the U-Net style decomposition.

A1: There might be some misunderstanding. In our study, we have conducted comparisons with various baselines that use U-Net, including both diffusion methods (TimeGrad, SSSD, and CSDI) and non-diffusion baselines (D$^3$VAE and PSA-GAN). All these baselines utilize U-Net architectures, which consist of multiple residual blocks with downsampling and upsampling to capture multiresolution information in the time series. Additionally, in our ablation studies (refer to $\underline{**Table 9 in Appendix F**}$), we also compared with a variant of the proposed mr-Diff which replaces the seasonal-trend decomposition with a U-Net style decomposition employing downsampling and upsampling. The results from these comparisons demonstrate the effectiveness of the seasonal-trend decomposition in mr-Diff.

Q2: In terms of computation complexity, the authors should also compare the model training time, especially with TimeDiff.

A2: As suggested, the table below shows the number of trainable parameters of different time series diffusion models and the corresponding training time (in ms) with different prediction horizons ($H$). As can be seen, the training of mr-Diff is faster than the existing diffusion models.

Q3: For inference, the authors stated that "mr-Diff is even faster than TimeDiff" which is also suspicious. All factors presented by the authors apply to TimeDiff as well. Specifically, TimeDiff can be regarded as mr-Diff with S = 1.

A3: There might be some misunderstanding. Note that TimeDiff is indeed NOT the same as mr-Diff with S = 1, as the conditioning network of mr-Diff is simpler than that of TimeDiff. In TimeDiff, a deep network with 10 convolution layers is used to map the lookback window for future mixup; while mr-Diff simply uses a linear mapping in the conditioning network (see Figure 3). This design choice helps to reduce the computational complexity. Hence, this explains why mr-Diff is faster than TimeDiff also on inference.

Q4: Why “while the seasonal-trend decomposition obtains both the seasonal and trend components, the focus here is on the trend.”?

A4: This is because the denoising process of mr-Diff proceeds in an easy-to-hard manner, and thus we focus on the trend to generate fine-to-coarse trends. This is also explained in the last paragraph in Section 4.1: ''as we use the diffusion model for time series reconstruction at various stages/resolutions (Section 4.2), intuitively, it is easier to predict a finer trend from a coarser trend. Reconstruction of a finer seasonal component from a coarser seasonal component may be difficult, especially as the seasonal component may not present clear patterns''.

Q5: Why “when X0 is used as in existing time series diffusion models, the denoising network may overfit temporal details at the finer level”?

A5: As mentioned in the second paragraph of Section 4.2, both the lookback segment $X_s$ and the generation target $Y_s$ are obtained from the same resolution level. Consequently, utilizing $X_s$ facilitates a more effective and straightforward reconstruction of $Y_s$. On the other hand, $X_0$ is obtained at the finest resolution level, and contains temporal details that may not be relevant to $Y_s$. Consequently, including $X_0$ in the denoising process of $Y_s​$ may lead to performance degradation due to overfitting of the unrelated details.

Q6: Minor: unlike existing $>$ Unlike existing that directly denoise $>$ that directly denoises

A6: Thanks for your careful reading. We will correct these typos in the next version.

I appreciate the thorough responses and explanations provided by the authors. Most of my comments have been addressed. I am not fully convinced that using a finer resolution $X_0$ with better details leads to overfitting unless we have better evidence. I will keep the same score and lean towards accepting the paper