Stochastic Diffusion: A Diffusion Based Model for Stochastic Time Series Forecasting

We would like to thank the reviewer for spending time and providing these comments. We are willing to provide more explanation for our method.

• Why is a Fully Connected Network (FCN) necessary in the Prior Encoder to project the sampled random variable $z_t$? Wouldn't it be possible to directly match the dimensions of parameters to latent variable? What is the rationale behind this approach?

Answer: The FCN with a non-linear activation function used in our model functions as a projector during the entire prior learning process. This design ensure flexibility and expressiveness of the model as it can learn non-linear relationships between the variables, leading to a richer latent space. Secondly, it is more effective to use a network for high-dimensional and multi-modal data comparing to matching the dimensions of variables. Finally, as a part of the deep learning model, embedding with FCN allowing end-to-end training and optimization. Thus, although it is true that we can directly match the dimension of the parameters to the latent variables, a learning approach should be a better option here. This is also a common design in existing works [1,2,3].

• "With equation (9),the prior distribution at... which enables it to (1)encode more informative representations,(2)better approximate the real data distributions, and subsequently,(3) benefit the time series data modeling." This statement lacks any actual explanation. Why does using the random variable as guidance confer these advantages?

Answer: We are willing to provide more details on the advantage of a dynamic prior distribution in our approach:

(1) Comparing to the common fixed prior (e.g. $\mathcal{N}(0,1)$), a step-wise dynamic prior allows the model to adapt its understanding of latent variables based on the historical data up to that point. This adaptability enables the latent variables to encapsulate relevant temporal dynamics and correlations specific to the period being modeled, thus encoding richer and more informative representations of the data.

(2) Most real-world time series data often exhibit non-stationary behaviors (changes in mean, variance, seasonality), a dynamic prior enables the model to adjust continually, maintaining alignment with the actual data distribution.

(3) Previous two points together contribute to a better time series modelling: As the model can quickly integrates fresh insights based on the dynamic, data-driven latent variable. The continual updating mechanism also helps in maintaining a high level of accuracy in data modelling, as the parameters are always optimized based on the latest available data.

Using random variables $z_t$ during time series modelling transforms the traditional approaches from merely analyzing temporal patterns to understanding and leveraging the underlying stochastic processes that generate the data. With the dynamic, data-driven priors, the model is responsive and adaptive to the most recent observation, leading to a more accurate data modelling process.

References:

[1] D. P. Kingma and M. Welling. Auto-encoding variational bayes. 2014.

[2] J. Chung, et al. A recurrent latent variable model for sequential data. 2015.

[3] A. Desai, et al. TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation. 2021.

We would like to thank the reviewer for these valuable comments. We now answer the reviewer's question.

• Can the authors discuss the limitations of the work in capturing highly-stochastic time series, or any other types of sequential data?

Answer: We provided a brief summary of the possible limitations and future works of the current model in Section F. We pointed out three possible limitations of the current model: (1) prior knowledge provided as condition in generative stage, (2) sequential backbone is a basic LSTM, (3) real-world validation. We are willing to go through these points in detail and provide more possible limitations and corresponding improvements:

(1) In our model, the prior knowledge is provided as a condition during the generative process. Although this is a common design at current stage (applied by most existing works on diffusion model which are cited in paper), further improvement is possible as a previous work investigated the directly usage of a learnt prior as the start point of the reverse diffusion process. This improvement may probably increase the efficiency of the diffusion process. On the other hand, the current model projects the learnt hidden units into corresponding prior knowledge using a MLP with non-linear activation function. A more robust architecture may possibly improve the prior knowledge learning comparing to this design.

(2) Our model uses LSTM as the sequential backbone, which lead to one limitation of the current model: as we designed it to model time series at each step, the local variance/uncertainty can be efficiently extracted. However, the long term features still relies on the LSTM to capture. Thus, as we explained in section 5.3, on long-period homogeneous datasets (e.g. Electricity), our model did not show superior performance comparing with baselines, such as MG-TSD or TMDM, which focus on these homogeneous datasets to develop certain mechanisms to extract temporal features (e.g. MG-TSD proposed to learn temporal features with multiple granularity). In the future, a combination of different sequential modules can be proposed to learn different features in time series data. For example, in Time-Transformer, a network consists of TCN and Transformer is proposed to simultaneously learn global and local features.

(3) As the current study is solely for research purpose, the model was only tested on limited datasets. To apply it for real-world tasks like human surgeries, more comprehensive testing and validations are required to guarantee the patients' safety.

(4) The current model apply parallelism and DDIM sampling strategy during the forecasting stage to enhance the sample speed. This strategy achieve the time efficiency with the cost of increasing memory requirement. In the future, a light-weight model design can be explored to achieve efficiency in both time and memory

We have update the Section F to include more details we provided here.

We would like to thank the reviewer for these valuable comments. We now answer the reviewer's questions point by point.

• "integration of the diffusion into the modelling process" vs."posthoc": unclear, isn't it more about recurrent prediction or some other model?

Answer: The main difference between our approach and the existing ones, as we mentioned in introduction, lies in the modelling stage. The existing methods solely use sequential networks (e.g. RNN, Transformer) during the modelling stage to learn the temporal features. The diffusion module is only applied to generate future values conditioned on the learnt features. Thus, the diffusion in these approaches is a posthoc style. However, in our model, the diffusion is integrate into modelling stage to help the sequential model extract the uncertainty and learn the multi-modal distribution of complex time series data from the observation. These lead to a better modelling of highly stochastic time series data, enabling the model to produce more accurate prediction of the future values.

We use the Table below to highlight the difference of the "integration of the diffusion into the modelling process" and the "posthoc":

• The figure does not completely fit to the text and the legend

Answer: We have update the figure to fit the text description and the legend.

• eq 11: is the complicated form needed?

Answer: Eq 11 formulates the reverse diffusion process of our approach and contrasts it to the one in normal diffusion models (provided in eq 3). However, as suggested by reviewer, we simplify its formulation to eliminate any redundancy and enhance clarity: $p_{\theta}(x_t^{n-1}|x_t^n,z_t) = \mathcal{N}(x_t^{n-1}; \mu_{\mathcal{C}}(x_t^n, n, z_t), \delta_{\mathcal{C}}(x_t^n, n, z_t))$, where $\mu_{\mathcal{C}}, \delta_{\mathcal{C}}$ are functions that compute the parameters of the Gaussian distribution. They integrate $z_t$ into corresponding parameters via function $\mathcal{C}$ (cross attention in our model).

• algorithm 1. There are two $z_t$ symbols as results from the two encoders.

Answer: We have update the Algorithm 1 to avoid any confusions. The corresponding step now become "Obtain $p_z, q_z$ and $z_t\sim q_z$ from equation (9), (10)".

• section C: please provide information about the Attention-Net which is crucial for the performance of the diffusion process.

Answer: We have updated section C to include more details about Attention-Net: The Attention-Net consists of a series of residual blocks adapted from existing works ([1,2,3], etc.). Each residual block contains two different attention mechanisms to generate original data: Firstly, a self-attention is applied to extract channel correlations for $x_t$. Then, a cross-attention combines the outputs of self-attention with the conditional latent $z_t$ to include the temporal dynamics, uncertainties and multi-modalities into the data prediction process. Finally, a feed forward network projects the outputs of residual blocks into the expected outputs - the predicted data. Through the backpropagation, the model will optimize the parameters in each part to learn the expected components.

In addition to the text description, we also include a figure to visualize the network structure. This can be found in Section C in updated paper.

References:

[1] A. van den Oord, et al. Wavenet: A generative model for raw audio. 2016.

[2] Z. Kong, et al. Diffwave: A versatile diffusion model for audio synthesis. 2021.

[3] J. L. Alcaraz and N. Strodthoff. Diffusion-based time series imputation and forecasting with structured state space models. 2023.

We would like to thank the reviewer for these valuable comments, we now answer reviewer's questions point by point.

• Why is only an MLP used for prior encoding? Would a more robust structure enhance the accuracy of the prior?

Answer: There might be some misunderstanding of our model's architecture, we would like to explain more details for a better understanding. The MLP (together with a non-linear activation function) in our model functions as a projector during the entire prior learning process. Overall, the recurrent model firstly extracts the hidden units $h_{t-1}$ at each time step. Then the MLPs with an activation function are applied to project the $h_{t-1}$ into corresponding parameters $\mu, \delta$ and further project parameters into final variable $z_t$ with demanded dimensions. Thus, more precisely, the prior knowledge is encoded by both the sequential model and the MLP, rather than solely by MLP. We notice that our description in Section C may cause this misunderstanding. We have updated the corresponding sentences to avoid any further confusion.

As to the network selection, we acknowledge the reviewer's suggestions on the model design: a more robust structure may further enhance the prior learning procedure. In our work, we are following previous works (VAE, VRNN, TimeVAE) [1,2,3] to use MLPs with activation function to project the hidden units and parameters into final latent variable $z$. We have added this valuable suggestion into our future work (Section F) and will try other model design in the future.

• What specific mechanisms contribute to the claimed fast sampling times?

Answer: The content on sampling time is briefly discussed due to the limited space. We are willing to provide more details about this part. The main strategy leads to our fast sampling results is the parallelism, as we design the model to generate $n$ samples simultaneously. On the other hand, we also apply the DDIM [4] method to reduce the reverse diffusion steps, enabling a faster sampling.

• Are the latent variables in the training process derived from Equation (9) or Equation (10)? The design and practical use of Equations (9) and (10) are not clearly explained, making it challenging to understand how and why these formulations are applied.

Answer: We notice the confusing part in the mentioned equations and their relations to the latent variable learning. We would like to provide more explanations to make these clear:

First of all, the equation (9) is used to compute the prior distribution of latent variable $z_t$, which is conditional on historical data $x_{1:t-1}$ and the corresponding latent variables $z_{1:t-1}$, $p_z(z_t|x_{1:t-1}, z_{1:t-1})$

In the meantime, equation (10) is used to compute an approximated posterior distribution of $z_t$, which now also conditional on the real data at current step $x_t$, $q_z(z_t|x_{1:t}, z_{1:t-1})$

The idea of variational inference is applied here to minimize the divergence between the approximate posterior $q_z$ and the true posterior $p(z_t|x_{1:t}, z_{1:t-1})$, in order to find the best approximation of the true posterior. However, due to the intractability of the true posterior, it is infeasible to directly calculate the divergence between approximation and the true one. Following the famous surrogate proposed by VAE [1], the evidence lower bound (ELBO) is applied here to indirectly minimize the original objective. This is includes a KL divergence between $q_z$ and $p_z$ and a likelihood of the observed data under the model. The variational inference effectively ensures that maximizing the ELBO is equivalent to minimizing the KL divergence between the approximate posterior and the true posterior.

We have updated several contents that may cause the confusion of latent variable learning, including the section 4.1 and the Algorithm 1.

References:

[1] D. P. Kingma and M. Welling. Auto-encoding variational bayes. 2014.

[2] J. Chung, et al. A recurrent latent variable model for sequential data. 2015.

[3] A. Desai, et al. TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation. 2021.

[4] J. Song, et al. Denoising diffusion implicit models. 2021