Transformer-Modulated Diffusion Models for Probabilistic Multivariate Time Series Forecasting

Q1:

Thank you for your insightful comment and suggestions. We intend to revise our paper to emphasize the following contributions: Firstly, as you mentioned, both components of TMDM originate from existing literature. However, amalgamating these techniques into a unified framework presents several technical challenges, such as effectively organizing the information captured by transformers and optimizing the entire framework using a single ELBO. Additionally, we bifurcate the probabilistic multivariate time series forecasting tasks into two parts: utilizing a well-designed Transformer to capture the mean of the MTS and considering covariate-dependence across both the forward and reverse processes within the diffusion model. We believe this amalgamation of the advantages from both models holds significant importance. Moreover, TMDM functions as a plug-and-play framework, seamlessly compatible with existing well-designed transformer-based forecasting models. This aspect represents another significant contribution of TMDM, bridging the gap between point estimates and distribution estimates. We believe this will reshape the community's perception regarding the relationship between point estimates and distribution estimates, as well as address practical application needs.

Despite PICP and QICE metrics originating from existing methodologies, they were initially introduced by us in the context of probabilistic multivariate time series forecasting tasks. As illustrated in Table 2 of our paper, D3VAE and TimeDiff were initially crafted for point-to-point forecasting tasks, with less emphasis on probabilistic forecasting. Consequently, their performance in distribution estimation (depicted in Figure 2) is inadequate, yet they exhibit competitive CRPS and $CRPS_{SUM}$ scores in Table 2. This underscores the insufficiency of existing metrics to handle the advancements in distribution estimation. Following these experiments, we conclude that the incorporation of QICE and PICP holds significant value. Hence, we advocate for future research to adopt these new metrics as standard evaluation criteria. Thank you once again for your valuable suggestion. We have revised and enhance our contribution based on your suggestions:

We summarize our contributions as follows: (1) In the realm of probabilistic multivariate time series forecasting, we introduce TMDM, a transformer-based diffusion generative framework. TMDM harnesses the representations captured by well-designed transformer-based time series models as priors. We consider the covariate-dependence across both the forward and reverse processes within the diffusion model, resulting in a highly accurate distribution estimation for future time series. (2)TMDM integrates diffusion and transformer-based models within a cohesive Bayesian framework, employing a hybrid optimization strategy, it serves as a plug-and-play framework, seamlessly compatible with existing well-designed transformer-based forecasting models, leveraging their strong capability to estimate the conditional mean of time series, facilitating the estimation of complete distributions. (3)In our experimental evaluation, we introduce two novel metrics for the probabilistic multivariate time series forecasting task: Prediction Interval Coverage Probability (PICP)and Quantile Interval Coverage Error (QICE). These metrics enhance the assessment of the uncertainty estimation prowess of probabilistic multivariate time series forecasting models. Evidenced by the state-of-the-art performance in four distribution metrics across six real-world datasets, TMDM shows effective potency in probabilistic MTS forecasting.

Q3:

We genuinely appreciate the time and effort you dedicated to a thorough reading of our paper. Your detailed review has been invaluable, and we apologize for any shortcomings in our references. Rest assured, we have diligently reevaluated all references to ensure they are brought up to date and meet the highest standards of academic rigor and accuracy. Your feedback is highly valuable, and we are committed to enhancing the quality and reliability of our work based on your suggestions. Thank you once again for your contribution to our research.

The updated references are listed as follows, and we'd appreciate your further feedback if you find additional refinances worth mentioning.

[1] Juan Lopez Alcaraz and Nils Strodthoff. Diffusion-based time series imputation and forecasting with structured state space models. Transactions on Machine Learning Research, 2022.

[2] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, pp. 4171– 4186, 2019.

[3] Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs M Bergmann, and Roland Vollgraf. Mul- tivariate probabilistic time series forecasting via conditioned normalizing flows. International Conference on Learning Representations, 2020.

[4] Lifeng Shen and James Kwok. Non-autoregressive conditional diffusion models for time series prediction. International Conference on Machine Learning, pp. 6657–6668, 2023.

[5] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2020.

[6] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. International Conference on Learning Representations, 2021.

[7] Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Doshi-Velez. Quality of uncertainty quan- tification for bayesian neural network inference. CoRR, abs/1906.09686, 2019. URL http: //arxiv.org/abs/1906.09686.

[8] Huangjie Zheng. Truncated diffusion probabilistic models and diffusion-based adversarial auto- encoders. Advances in Neural Information Processing Systems, 36, 2022.

Q4:

Thank you for your comment and suggestion. Due to the full text page limit, details are omitted from eq(9) to eq(10), we have provided in the Appendix will provide a more detailed explanation of this section to enhance clarity. In the context of a Bayesian generative model, we approach this matter from a generative standpoint. Initially, we create a sample $z$ drawn from the standard normal distribution $\mathcal{N}(0, \boldsymbol{I})$. Subsequently, we employ an MLP represented as $\mu_z$ to derive the distribution parameters of $P(y{0:M})$. This methodology enables the sampling of $y_{0:M}$ from the distribution $P(y_{0:M})$. It's important to note that the dimension of $z$ is $\mathbb{R}^{N \times M}$, with $N$ set as 512 across all datasets.

Alternatively, considering this segment as a hierarchical VAE, the neural networks representing $\mu_z$ and $\sigma_{z}$ function as parts of the encoder, while $\mu_z$ serves as part of the decoder. In our paper, these three neural networks are implemented using MLPs with a dimension of 512.

For a fair comparison, we utilized the STransformer in a setting similar to PWN as the basic Transformer. When comparing TMDM+STransformer with PWN, TMDM shows improvements across most datasets. Furthermore, using PWN as the basic Transformer, TMDM+PWN yielded the best results among the three experiments (PWN, TMDM+STransformer, and TMDM+PWN)

Q1:

Thank you for your comment and suggestion. The variation in the $CRPS_{sum}$ results of the baseline models compared to the published results is primarily due to differences in the experimental settings, specifically concerning the history length and prediction length. We have included the experiments on the settings in TimeGrand and CSDI in our paper, and the results can also be found as follows:

CSDI occasionally matches or slightly surpasses TimeGrand, yet TMDM consistently outperforms both models across various datasets and metrics. TMDM showcases superior performance in probabilistic forecasting, reflected in its lower QICE and CRPS-sum values. These results emphasize TMDM's significant advancements in predictive accuracy and distributional modeling compared to TimeGrand and CSDI across diverse datasets and evaluation metrics.

Q2:

Thank you for your insightful question. Evaluating model performance with time-series data at different granularities is of significant importance in real-world applications. We believe TMDM can excel in such scenarios for the following reasons:

1. Similar to most time series forecasting models [4,5,6], we incorporate the actual timestamps as learnable time embeddings for each data point. Leveraging a well-designed Transformer, we can effectively capture the temporal correlations within the data. This design ensures that TMDM can adapt to various granularities of time-series data.

2. Within the diffusion model component, we also account for the time embedding in the data. This allows the model to generate multivariate time series with information from these embeddings, accommodating time series data at different granularities.

3. As shown in Table 1 in our paper, the selected datasets covered different granularities ranging from 10 minutes to 1 day, and TMDM demonstrated competitive performance across all databases. This confirms the model's ability to handle situations where time-series data is available at different granularities.

4. To further evaluate TMDM's ability to handle varying time-series granularities, we conducted an experiment where we randomly removed $D$ data points from a given time series $y_{0:M}$ and used this modified dataset to test TMDM with the same settings as described in the paper. This challenging experiment simulates scenarios where the time intervals in multivariate time series vary, making it a rigorous test of the model's performance under changing granularities.

In the presented table, TMDM-60 refers to our model with a prediction length of 192 + 60, where we randomly exclude 60 samples from the data, introducing random time intervals between consecutive points. In this scenario, TMDM is tasked with forecasting multivariate time series (MTS) with varying granularities based on temporal embeddings. Remarkably, the results from the 30 and 60 settings display similar scores to the original setting, showcasing the efficacy of TMDM in handling time-series data with diverse granularities.

[4] Liu, Y., Wu, H., Wang, J., & Long, M. (2022). Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35, pp. 9881-9893.

[5] Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, pp. 22419-22430.

[6] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2021, May). Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35, No. 12, pp. 11106-11115.

Q1:

Thank you for your valuable comments and suggestions. In light of your feedback, our paper has been revised to encompass more state-of-the-art baselines, such as cST-ML [1] and DAC-ML [2], both of which utilize VAE for dynamic capture. These new inclusions serve to deepen our analysis and broaden the scope of our research. The experimental results, now integrated into the updated version, are presented as follows:

The results in the above Table show that the intricately structured CNN-RNN and Bayesian framework in cST-ML [1] led to an 8% average reduction in QICE compared to the VAE model. DAC-ML [2], which builds upon cST-ML by incorporating model adaptation capabilities, achieved a 15% reduction in QICE relative to the VAE-based approach. However, it's noteworthy that the proposed TMDM still surpasses both cST-ML [1] and DAC-ML [2] in performance, despite their enhancements over the standard VAE. The reasons for this significant outperformance by TMDM are detailed as follows:

1. Transformer-based architectures have shown remarkable success across a wide range of tasks. However, models like DAC-ML and cST-ML, which are based on CNN-RNN structures, face challenges in capturing long-term correlations. In the field of time series forecasting, specialized transformers are developed to address the nuances of multivariate time series. These specialized transformers often outperform generic transformer models. TMDM leverages these specifically tailored transformers, thereby gaining a significant edge over RNN-based methodologies.

2. The diffusion model, recognized as a state-of-the-art deep generative model, has been extensively validated for its enhanced generative capabilities over basic VAEs in numerous studies. TMDM integrates a conditional diffusion generative process, which facilitates accurate distribution forecasting in multivariate time series. This attribute equips TMDM with a substantial advantage in performance over other VAE-based models like cST-ML.

3. TMDM effectively utilizes transformers to distill crucial insights from historical time series data. These insights serve as prior knowledge, enabling the model to capture covariate-dependence effectively during both the forward and reverse phases of the diffusion process. This synergistic integration of transformers and diffusion models positions TMDM at the forefront in probabilistic multivariate time series forecasting.

4. TMDM's design transcends a mere combination of two modules. Instead, it integrates these models within a cohesive Bayesian framework, employing a hybrid optimization strategy. As detailed in Equation 15 of our paper, the first term guides the denoising model to predict uncertainty while subtly adjusting the conditional generative model to provide a more appropriate conditional representation. The second term facilitates the generation of improved conditional representations by harnessing the capabilities of a well-designed transformer. The two parts of the model adapt to each other within a hybrid framework, further enhancing the model's performance.

[1] Zhang, Y., Li, Y., Zhou, X., & Luo, J. (2020, November). cST-ML: Continuous spatial-temporal meta-learning for traffic dynamics prediction. In 2020 IEEE International Conference on Data Mining (ICDM), pp. 1418-1423.

[2] Zhang, X., Li, Y., Zhou, X., Mangoubi, O., Zhang, Z., Filardi, V., & Luo, J. (2021, December). DAC-ML: domain adaptable continuous meta-learning for urban dynamics prediction. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 906-915.